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EXECUTIVE SUMMARY 


The Aeroflightdynamics Directorate (AMRDEC) conducted a simulation to examine the 
performance of the Comanche Contact Analog world-referenced symbology displayed on the 
Comanche HIDSS when compared with a compressed symbology design similar to that specified by 
the former MIL-STD 1295. Six experimental test pilots flew one modified ADS-33 maneuver (hover 
turn, bob-up), an unusual attitude recovery, and two terrain flight tactical tasks in the NASA Vertical 
Motion Simulator (VMS). Analysis of the pilot objective performance data and subjective data 
showed the following results. 

Objective test results showed that 1295 symbology yielded more rapid maneuvering in the hover 
turn bob-up than Contact Analog symbology. The average margin of difference in the time to 
complete the maneuver was approximately two seconds, which was statistically significant. There 
were no significant differences measured between symbology sets with respect to altitude or position 
performance measures for all other maneuvers. The NOE target ID task data showed improved 
accuracy in determining heading to target when using Contact Analog over MIL-STD- 1295. 

Subjective test results, including handling qualities ratings (HQRs) and NASA-TLX workload 
ratings, showed small but consistent advantages of 1295 symbology over Contact Analog for most 
parameters. For the bob-up maneuver, 1295 symbology handling qualities were rated “Desired” for 
lateral position error and time to complete whereas Contact Analog was rated adequate. The average 
HQRs for all other maneuvers were rated the same for both symbology sets (see summary chart. 
Table 1). 

Pilot comments and the results of an online questionnaire more strongly favored 1295 over Contact 
Analog. Repeated comments from all six pilots led to a focus on design issues with six Contact 
Analog symbols. Those symbols were the heading tape, horizon line, radar altitude six-second 
predictor, the position of the torque symbol, the absence of a hover position cue, and the widespread 
positioning of the symbology to the outer edges of the display field-of-view. Pilots rated the present 
design of three of the six symbols as having safety-of-flight implications. Those symbols were the 
heading tape, horizon line, and six-second predictor. 

To summarize, test results showed no objective data that would warrant restricting experimental test 
pilots from flying constrained tasks. Small but consistent advantages were recorded for the MIL- 
STD- 1295 symbology design when executing ADS-33 constrained tasks. No general performance 
differences were recorded for the operational maneuvers (NOE, contour flight modes) except for the 
azimuth-to-target task, which favored the Contact Analog heading tape design. There were 
consistent and strong pilot comments supporting the MIL-STD- 1295 design over Contact Analog in 
both this current simulation and in Comanche Sim I. Three Contact Analog symbols warrant 
modification and further evaluation to mitigate safety-of-flight implications noted by participating 
test pilots. 

A plan for symbology redesign and testing was developed. However, the Comanche program was 
cancelled by Department of Army in February 2004. 
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SECTION 1. SIMULATION TEST PLAN 


1.1 Simulation Objectives 

The primary purpose of this piloted simulation was to evaluate the effectiveness of two Helmet- 
Mounted Display (HMD) symbology sets. Those were the RAH-66 Contact Analog symbology, and 
a more traditional MIL-STD-1295 (ref. 7)-based design. A secondary purpose was to explore three 
Contact Analog-based hybrid designs. Each symbology set was examined while pilots executed one 
handling qualities task, a situational awareness task, and a mission-oriented scenario in the NASA 
Vertical Motion Simulator (VMS). Emphasis was on quantifying pilot performance, workload, and 
overall effectiveness of the heading tape, attitude, altitude, and hover symbology. This simulation 
was built upon a prior year 2002 experiment conducted in the VMS (fixed-base) with similar 
objectives (ref. 11, AHS 2002 paper). Significant differences from the prior simulation included: 
a) Use of the VMS large amplitude motion cueing; b) Use of the Automatic Flight Control System 
(AFCS) as the primary flight control mode throughout; c) Evaluation of hybrid Contact Analog 
symbology sets; d) Inclusion of unusual attitude recovery and mission-oriented terrain flight pilotage 
tasks. 


1.2 Background 

The RAH-66 Comanche Scout/ Attack Helicopter was slated to be the first Army helicopter to use an 
HMD as the primary flight display (PFD). The symbology design concept described in the Pilot 
Vehicle Interface Mechanization Specification (PVIMS) (ref. 3) incorporates an inertially referenced 
system known as “Contact Analog,” wherein symbols appear to overlay the real-world objects they 
represent. Sikorsky has described the Contact Analog design philosophy as maintaining a “Gestalt,” 
where visual, vestibular, and proprioceptive cues remain in agreement. That is, in the virtual world 
of the HMD, symbols behave like the real-world parameter they represent. Earth-referenced symbols 
appear to remain fixed to their referents outside of the aircraft. Aircraft-referenced symbols appear 
to move with the aircraft. Head-referenced symbols appear to move with the head. Sikorsky 
symbology designers believe that decoupling any of these inputs from each other has potential for 
disorientation. Contact Analog symbology is based on the notion of fidelity to movement, behavior, 
and interrelationships of real-world referents. Dynamics, location, and behavior of a symbol 
represent its corresponding source of information in contact (visual) flight. 

The Helmet Integrated Display Sight System (HIDSS) has recently been introduced into the 
Comanche flight test program. Initial tests of the primary flight symbology were flown by Army 
pilots in the Sikorsky Engineering Development Simulator (EDS) and in the Comanche Portable 
Cockpit (CPC) using a surrogate HMD prior to introducing the HMD to the flight test vehicle. These 
initial tests surfaced issues regarding the usability of the HMD primary flight symbology. These 
concerns were documented in a White Paper published by the TRADOC Systems Manager (TSM), 
Comanche, in August of 2001 (ref. 5, TSM White Paper). The issues were related primarily to the 
implementation of the heading tape and artificial horizon line, although other symbols were 
identified with a lesser degree of concern. 
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A fixed-based simulation was conducted at Ames Research Center in January-February 2002 to 
address the TSM's concerns. The primary objective of this effort was to examine the functionality of 
the Contact Analog heading tape when compared to a traditional HMD presentation designed around 
the former MIL-STD-1295, Human Factors Engineering Design Criteria for Helicopter Cockpit 
Electro-Optical Display Symbology. Six pilots (3 Apache rated) flew four test maneuvers. There 
were two hover tasks, and two up-and-away tasks. Three of the four maneuvers were modified 
Aeronautical Design Standard 33 (ADS-33) (ref. 1) tasks and the fourth was selected from the Army 
Aircrew Training Manual (ATM) (ref. 4). Two flight control modes were flown as independent 
variables. The modes were Velocity Stabilization (VelStab) and the Automatic Flight Control 
System (AFCS) mode. 

The 2002 simulation results validated many of the TSM's concerns. Data showed small but 
statistically significant improved pilot performance using MIL-STD-1295 symbology over Contact 
Analog. Larger performance differences were noted between flight control modes with VelStab 
outperforming AFCS. NASA-TLX results showed a lower workload for MIL-STD-1295 symbology 
over Contact Analog in both flight control modes. Readability of the Contact Analog heading tape 
during rapid turning movements was an issue. Handling qualities ratings indicated a need for 
improvement associated with the design and implementation of selected Contact Analog symbology. 
One conclusion of this study was that additional simulation experiments should be conducted to 
examine the Contact Analog symbology design in a more mission-oriented, operational setting. The 
results of this simulation were presented at the 58th Annual American Helicopter Society (AHS) 
Forum, June 2002 (ref. 11). The simulation described below is a follow-on to the 2002 simulation 
effort. 


1.3 Simulation Overview 

The Aeroflightdynamics Directorate (AFDD) conducted this experiment in the VMS driven by a 
high-fidelity Comanche helicopter math model integrated with a head-tracked HMD. The simulation 
results were to be used as an independent data source for the Aviation and Missile Research and 
Development Engineering Center (AMRDEC) Comanche airworthiness evaluations. The primary 
focus of the simulation was to collect and analyze performance data and to isolate issues regarding 
readability, usability, and pilot performance using the Contact Analog HMD symbology as a 
primary flight reference. 

The experiment used a MIL-STD-1295-based symbology design and functionality as the baseline. 
Contact Analog functionality and performance was compared to the baseline. The simulation 
involved one constrained ADS-33 task, one situational awareness task, and two tactical tasks. This 
simulation examined the “Contact Analog” design philosophy to determine if stated benefits could 
be utilized without adversely affecting crew performance. The simulation attempted to quantify the 
issues with performance data from various performance measurement tools. Quantitative data 
collected were supplemented with handling-qualities ratings and pilot comments as well as 
situational awareness ratings. 
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1.4 Hybrid Symbology Sets 


There were three Contact Analog-based alternate designs developed and tested that represented 
hybrids (mixtures) of Contact Analog and MIL-STD-1295 symbology. These hybrids were 
developed based on findings from the previous 2002 Comanche symbology simulation, that 
suggested a hybrid of the two symbology sets may improve pilot performance. The hybrid designs 
were strongly influenced by pilot comments from that simulation and by designs that were being 
informally considered by the user community at Ft. Rucker. Although the three hybrid symbology 
designs were compared to the baseline, this analysis was considered as a separate science and 
technology effort and not as a part of the primary test matrix. 


1.5 Method 


1.5.1 Participants 

Six experimental test pilots were used as subjects to support the experiment. Four were from the 
Aviation Technical Test Center (ATTC) and two were local Army and NASA pilots. Pilots had 
flight experience in multiple airframes, each logging an average 351 1.83 total flight hours ( n = 6) 
(fig. 1). The low-time pilot had over 2000 flight hours. All pilots were instrument rated. Five of six 
pilots were night vision goggle qualified. Pilot selection criteria required pilots with minimal 
exposure to MIL-STD 1295 symbology in order to minimize any bias for 1295 over Contact Analog 
symbology. Consequently, few pilots reported prior HMD experience. Five of the six pilots had the 
UH-60 designated as their primary aircraft and one had CH-47 as primary. 



Pilot ID 

Figure 1. Summary of Test Pilot Flight Experience. 
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1.6 Experimental Design 


A 2 x 2 x 4 (symbology x block x trial) factorial, within-subjects experiment with repeated measures 
was conducted. Symbology was tested across two levels: Contact Analog and compressed. 
Symbology conditions were blocked with counterbalanced presentation, such that pilots completed 
all maneuvers in one symbology set before training and testing in a different symbology. Maneuvers 
were blocked with counterbalanced presentation, such that all maneuvers were completed once 
before a block repetition. Four repetitions of each maneuver were executed within a single block. In 
sum, pilots completed 8 hover turn bob-ups (BU), 8 unusual-attitude recoveries (UA), and 2 tactical 
missions (TM) in each of the symbologies (fig. 2). 


BLOCK 1 

BLOCK 2 

task 1 

task 2 

task 3 

task 1 

task 2 

task 3 

BU x 4 

UA x 4 

TM 

BU x 4 

UA x 4 

TM 


Figure 2. Test Matrix (repeated for Alternate Symbology Set). 


Tactical missions contained 2 flight profiles: contour and NOE. Contour flight always preceded 
NOE. Both contour and NOE flight segments were completed once per block in each symbology 
condition (2 total). Contour flight mode consisted of two commanded airspeed changes (increase, 
decrease) and two commanded altitude changes (ascend, descend). RMSE on the non-commanded 
parameter was calculated in order to test a hypothesis on incorrect control inputs due to placement of 
airspeed and altitude symbology. Time to complete the command was also measured. 

NOE flightpaths consisted of an equal number of waypoints (n = 4) with two targets located within 
40-deg FOV and two targets outside, for a total of 4 targets per flight. Time between target detection 
and pilot annunciation (azimuth to target) was recorded. Accuracy of pilot annunciation of azimuth 
to target was analyzed. 

The dependent measures varied by maneuver and correlated with parameters outlined in ADS-33. 
Separate 2x2x4 (symbology x block x repetition) analyses of variance (ANOVAs) were conducted 
by maneuver and parameter. For the tactical missions, contour and NOE data were analyzed 
separately. Subjective dependent measures collected were NASA-TLX ratings (Hart & Staveland, 
1988), measuring perceived workload and situational awareness, and aircraft handling-quality 
ratings (Cooper & Harper, 1969) (ref. 10) for hover bob-up maneuvers only. Additional subjective 
ratings on specific display items (i.e., horizon line reference) within a symbology set were also 
collected. Ratings given assessed the usefulness of the display item and ease or difficulty to control 
related aircraft axes. 

Finally, 3 hybrid symbologies were presented to the pilots at the end of the primary experiment to 
collect additional data. Pilots executed a hover turn bob-up maneuver in each hybrid condition. 
Symbologies were presented in counterbalanced order. Four repetitions were completed in each 
symbology with initiated turn direction counterbalanced (left, right or right, left). In total, data from 
12 hover turn bob-up maneuvers were collected. Means and standard deviations of the data were 
calculated for the maneuver from the tolerance parameters provided. 


6 


1.7 Simulation Design 


1.7.1 Flight Control System Models 

The Comanche core Primary Flight Control System (PFCS), mission PFCS, and the Automatic 
Flight Control System (AFCS) were primarily used in this simulation. The hover bob-up maneuver, 
however, was flown in both AFCS and Velocity Stabilization (VelStab) with the Position Hold 
selectable mode engaged. Altitude Hold was not used. The intent was to contrast the workload 
differences in maintaining hover position in AFCS mode only when compared with Position Hold 
engaged. The choice to fly the majority of the experiment in AFCS was based on the desire to place 
more emphasis on the "hand flying" aspect of the available flight control systems. The results of 
Comanche Simulation I in 2002 indicated AFCS to be the most demanding flight mode on pilot 
performance and produced the most significant test results. 

The Comanche math models were loaded on the VMS host computer. The flight control system 
software used in the prior Comanche VMS simulation was used for this experiment. The software 
for the flight control system had been supplied and checked out by Boeing engineers to support 
Comanche Sim I. It was based on Operational Flight Program (OFP) 2D. As of April 03, OFP 2E 
was presently in flight test. Boeing engineers stated there were no changes from 2D to 2E that would 
impact the HMD symbology study. A validation test card was developed that specified specific 
maneuvers and procedures be flown that would ensure the vehicle math model, flight control system 
software, and the symbology performed in accordance with dynamic check cases and the 
performance parameters documented in the PVIMS. The Comanche Sim II project pilot flew this 
test card both in fixed-base and on full motion in the VMS F-Cab to validate the flight control 
system and symbology performance. An independent validation was also performed, which is 
described later in this report. 

1.7.2 Symbology Sets 

The Comanche Contact Analog PFD symbology graphic display and the associated drivers were 
identical to Comanche Sim I. The MIL-STD-1295 symbology and drivers from the previous 
simulation were also repeated in this simulation. The 1295 symbology was driven using the 
Comanche drive-laws to remain compatible with the Comanche flight control system. The 
symbology drivers were linked with the Comanche math model and control system models as 
required for data exchange. In addition to the above, the SimLab support staff developed three 
hybrid Contact Analog symbol sets. One incorporated the standard Contact Analog heading tape, 
which was screen-fixed in the roll axis, but remained referenced to the pilot’s head movement as 
well as aircraft heading. The second hybrid incorporated a screen-fixed, compressed MIL-STD-1295 
heading tape that was not head-referenced. The third hybrid was the standard Contact Analog 
symbology set with modified drive laws for the radar altimeter six-second predictor symbol. No 
weapons, tactical situation, or systems management symbology was displayed for this simulation. 
The symbology was displayed on a Kaiser ProView 50 HMD supplied by the Comanche PM from 
the Comanche Portable Cockpit. The Comanche Test Team used the procedures developed during 
the prior simulation to check out the symbology and drivers during simulation development. 

1.7.3 Symbology Description 

Figures 3 and 4 present both graphics and a short narrative description of the Contact Analog 
symbology design. 
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Alternate Sensor Heading Digital Command Heading 



Figure 3. Contact Analog Symbology Set. 


1.7.4 Contact Analog Heading Tape Description 

The Contact Analog implementation of the heading tape is illustrated below. Key features of the 
symbology are described in bullet form. The reference for the description is the Pilot Vehicle 
Interface Mechanization Specification (PVIMS) for Comanche (Rev. F, 7/99). 
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Figure 4. Contact Analog Heading Tape. 


• The HMD heading tape symbol is earth-stabilized in roll and azimuth, screen- stabilized in 
elevation, and moves across the display in relation to changes in heading of the aircraft and the 
pilot’s head. 

• The heading tape remains aligned with the real-world horizon. 

• When the A/C is flying straight and level, the heading tape appears to the pilot as a continuous 
360° tape around the aircraft. As an example, if the aircraft is heading due north and the pilot 
looks 90° to the right, “E” appears at the center of the heading tape. Approximately 40° remain 
in the instantaneous field of view (FOV). 
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• The heading tape has numerals at 10° increments (i.e., 10°, 20°, 30°, etc.) and has hash marks at 
5° increments in between (i.e., 5°, 15°, 25°, etc.). Alphanumerical characters (N, S, E, W) are 
displayed at the cardinal headings. The heading tape is removed in the de-clutter mode. 

• The heading is referenced to Magnetic North. 

• The A/C heading reference appears as an I-Bar that is earth-stabilized in roll, screen- stabilized in 
elevation, and aircraft-stabilized in azimuth. The steering cue symbol (upward-pointing caret) is 
stabilized like the I-Bar symbol and indicates commanded heading (desired heading to next 
waypoint) and moves horizontally across the display below the heading tape. 

• Aircraft heading is displayed digitally in the top center of the HMD display beneath the heading 
tape and is screen-fixed in that location. The digital readout is boxed when heading hold is 
engaged. 

• The artificial horizon line has occlusion priority over the heading tape when the head is tipped 
down enough to bring the two symbols into proximity. 

Figures 5 and 6 present both graphics and a short narrative description of the MIL-STD-1295 

symbology design. 


Alternate Crew 



Figure 5. MIL-STD-1295 Symbology Set. 
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1.7.5 Compressed Heading Tape Description 


W 30 33 360 3 6 E 



Figure 6. Compressed Heading Tape. 


• The heading tape is screen-fixed and displayed at the top area of the HMD. Screen- fixed means 
it appears as if painted on the helmet visor and moves with the pilot's head. 

• It is an analog moving scale that presents magnetic heading with a total range of 360°. A total of 
180° are in the instantaneous FOV at all times. 

• The scale is incremented every 10° and the major cardinal headings are alphanumerically 
labeled. 

Figures 7 and 8 present both graphics and a short narrative description of the hybrid symbology 
designs. 

1.7.6 Hybrid 1 Symbology Set Description 



• Standard Contact Analog symbology set with heading tape modified to be screen-fixed in pitch 
and roll axis 

• Remains head-tracked and aircraft-tracked in the yaw axis 
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1.7.7 Hybrid 2 Symbology Set Description 


Screen Fixed 



Figure 8. Hybrid 2 Symbology. 


• Compressed heading tape replaces Contact Analog heading tape 

• Tape is screen-fixed in roll and pitch axis (gyro- stabilized, not true earth referenced) 

• Appears as if painted on the helmet visor 

1.7.8 Hybrid 3 Symbology Set Description 


Modified 6 sec Predictor Drive Laws 



Figure 9. Hybrid 3 Symbology. 
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Standard Contact Analog symbology (fig. 9) 
Modified 6-second predictor drive laws 






1.8 Simulation CAB and Hardware 

The simulation utilized the VMS F-CAB as the cockpit. It was initially set up in the fixed-base lab 
of the VMS facility for checkout purposes before being moved onto the motion-based beam. The 
cockpit was configured for a single pilot with a sidearm controller and collective (fig. 10). A helmet 
mounted display and head-tracker provided the primary flight information to the pilot. A single 
panel-mounted Horizontal Situation Display (HSD) with a moving map was installed. The set-up 
was not intended to be a duplication of the Comanche cockpit. This austere arrangement provided 
only the controls and displays necessary to conduct the simulation. A detailed description of the 
controls and displays follow. 



Figure 10. General Cockpit Arrangement. 


1.8.1 Controllers 

A four-axis SAC provided longitudinal and lateral cyclic control, yaw axis control, and limited 
heave control. This controller (fig. 11) was loaned from the Comanche Portable Cockpit (CPC) and 
was a full emulation of the Comanche SAC, but designed for simulator use only. 

An Apache collective control was used in the simulation (fig. 12). Although it was not a Comanche 
collective stick, the functionality was matched as closely as possible to the Comanche. The 
collective was back-driven and had a displacement matching the Comanche specification, and 
control forces were matched as much as possible. The switches on the SAC head were used to 
support turning event markers on and off. 
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The sidearm controller (SAC) and collective were installed and positioned according to 
specifications provided by Sikorsky and were identical in arrangement to the 2002 Comanche VMS 
simulation. 



Figure 1 1 . Comanche 4- Axis Controller. 



Figure 12. Collective Control Head. 
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1.8.2 Helmet-Mounted Display 

The simulation utilized a ProView 50 HMD manufactured by Kaiser Electro-Optics on loan from the 
Comanche Portable Cockpit (fig. 13). The ProView 50 is a biocular display with a VGA (600 x 480 
@ 60 Hz) active-matrix, liquid-crystal display. The field-of-view was 28° vertical by 49° horizontal 
with 25° overlap. 

This display had a lower resolution and reduced FOV than the Comanche HIDSS system. The 
HIDSS display was to have active matrix LCDs, with SXGA resolution (1280 x 1024) and the field 
of view is 35° vertical by 52° horizontal with a minimum 17° overlap. 

The same ProView 50 HMD was used to support Comanche Sim I. It is important to note that 
1) minimal negative feedback was received by evaluation pilots on this matter during the 2002 
Comanche simulation, and 2) the diminished FOV and resolution was constant across all experiment 
manipulations and therefore did not bias comparisons. The contrast of the FLIR sensor scene as 
viewed through the HMD produced a Level 2 usable cue environment (UCE 2), and resulted in a 
larger dependence on the symbology than the outside scene to complete the maneuvers. This was 
considered a desirable affect. Standards for the maneuver were based on a UCE 2 visual scene. 

The symbology images for the HMD were generated by an SGI PC computer. The symbology 
overlaid the outside scene images produced by the image generator when combined in the HMD. 


Comanche HIDSS 



Kaiser ProView 50 



Figure 13. Helmet-Mounted Display. 
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1.8.3 Head Tracker 

The simulation utilized a Polhemus "Fasttrack" magnetic head tracker (fig. 14). The magnetic sensor 
was mounted to the cab above and to the rear of the pilot’s head. The transmitter was mounted to the 
top of the HMD. A thorough mapping of the head-tracker envelope was accomplished during the 
simulation set-up. A method for boresighting the head tracker was installed to ensure each pilot’s 
head position was in the head-tracker box prior to each pilot’s first flight of the simulation session. 



Figure 14. Polhemus Head Tracker. 


1.8.4 Image Generator for the HMD 

An Evans and Sutherland Image Generator, (CGI) Model 3000 (ESIG 3000), generated the FLIR 
image visual display for the HMD. The database used for the simulation was an in-house database of 
the Monterey Bay area. A terrain map of the database is shown in figure 15. The simulation 
evaluations were conducted using the FLIR mode only. 

1.8.5 Panel-Mounted Display 

The simulation used a single panel-mounted display. The display was an 8” x 8” flat panel. It was set 
up as a moving map depicting terrain relief, flight route, and aircraft present position with track-up. 
This display was only used for the tactical maneuvers (contour and NOE) and provided the pilots 
with additional situational awareness cues. 
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Figure 15. Plan View of Terrain Database. 


1.9 Simulation Cab Performance Validation and Checkout 
1.9.1 Math Model Performance Validation 

The Comanche math model was validated during Comanche Sim I in 2002. During that experiment 
it was checked out by the VMS engineering staff along with on-site assistance from a Boeing flight 
controls engineer using check cases supplied by Boeing. The model is an accurate, non-linear, large 
angle, free flight math model of the Comanche airframe, engine, drive-train, and flight controls. No 
changes were made to the model since the 2002 simulation experiment. 

The model was implemented during the pre-simulation ground checkout phase of this experiment in 
the F-Cab cockpit. The controllers were set up according to the Sikorsky design specifications. Plots 
of the controller characteristics were made for documentation. The AFDD project pilot checked out 
the combination of math model and controllers by flying a myriad of tasks from a pre-test matrix 
that validated performance against the PVIM specifications. The operation of the head tracker, 
visual system, and HMD were also verified during these trials. 

The F-Cab was then moved to the motion-base beam and the same checkout was flown while the 
cab was in motion. Motion tuning trials of the flight maneuvers were flown to adjust motion cues as 
necessary to ensure they were appropriate for the maneuvers flown during the test. 
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Once all local validation checks were completed, an Army test pilot from the Comanche Combined 
Test Team (who had flown the actual aircraft) performed verification flights in the VMS. He flew 
specific maneuvers and measured the results against the design specification for the actual aircraft. 
He also provided subjective comments concerning the handling qualities of the simulator in a written 
after-action report. Deficiencies were recorded and analyzed by the simulation team. All deficiencies 
that would affect the results of the experiment were fixed. 

A visiting Sikorsky human factors engineer (also a test pilot) verified the symbology drivers and 
visual performance. The process included a look at the design, movement, and functionality of each 
symbol, comparing performance to the most current version of the PVIMS design specification. 

Only minor deficiencies were noted and were corrected immediately. 
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SECTION 2. SIMULATION EXECUTION 


2.1 Training and Test Procedure 

The simulation was performed over a period of four weeks. Groups of one or two pilots were 
scheduled each day throughout the test period. Each pilot spent four to five days completing training 
and the test requirements. 

Test pilots were given documentation on all maneuvers and performance parameters in advance of 
testing. Both classroom instruction and simulation cab familiarization occurred prior to practice of 
the evaluation maneuvers. Each pilot flew a series of specific maneuvers using each of the 
symbology sets (Contact Analog, compressed, or hybrid). The maneuvers are detailed in the 
Evaluation Flight Task section below. Practice runs were flown prior to data runs for each 
maneuver. Each maneuver, except the tactical maneuver, was flown in both directions, left and right. 
Each pilot completed a criterion task to a desired performance standard before data collection was 
initiated. 

After executing an evaluation maneuver, pilots completed a Weighting of Rating Scales (Appendix 
D). NASA-TLX subscales were presented in pair-wise comparisons for the purpose of weighting the 
scale that was perceived to more significantly contribute to workload for a particular task. Upon 
completion of data trials for a maneuver, pilots rated workload on each of the subscales. Weighted 
overall workload scores were calculated for each pilot by applying the scale weights to the raw 
subscale ratings for a maneuver. 

Once testing within a symbology set concluded, pilots' subjective ratings on the effectiveness of the 
display symbology were collected in an online questionnaire. Specific symbols (i.e., horizon line 
reference) were rated separately for usefulness and ease or difficulty to control relative to aircraft 
axes. Additional pilot comments were collected in post-experiment debriefings. 

The simulation project pilot conducted all pilot training. The training sessions involved both ground 
school and simulator flight training. During ground school the pilots were presented an overview of 
the functionality of each symbol in each of the symbol sets. Separate sessions were taught for 
Contact Analog and MIL-STD-1295 just prior to data collection for that symbology. Training also 
included an explanation of the flight controls and flight control modes, and an overview of the flight 
test maneuvers and standards. Subject pilots were given instruction during structured familiarization 
flights in the VMS. Interaction of the flight control inputs and the symbology response was 
experienced. Training flights in a specific symbology set preceded the data runs requiring the use of 
that particular symbol set. Once the pilots were comfortable with the controls, displays, and 
symbology, they were required to fly an evaluation task to a specified performance standard before 
being allowed to proceed to the flight-test phase. A training syllabus is presented in Appendix A. 
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2.1.1 Evaluation Maneuvers 

The maneuvers flown during this experiment and shown below were intended to be realistic tasks 
that would be performed by Army pilots consistent with the Comanche Scout/ Attack mission. These 
tasks and performance standards were validated during the checkout phase of the simulation by 
project test pilots. An Army experimental test pilot from the Comanche Combined Test Team 
further independently validated the test maneuvers. 

The simulation utilized two levels of flight tasks to evaluate the effectiveness of the Comanche 
symbology. The levels included both a constrained handling qualities maneuver as well as a less 
constrained emergency procedure and tactical terrain flight profiles. The handling qualities 
maneuver was a hover turn bob-up, which is a modified ADS-33 task. The unusual attitude recovery 
was a discrete task developed from the Attack Helicopter Aircrew Training Manual that was 
intended to measure the usability of the heading tape, horizon line, altitude, and other symbology in 
maintaining situational awareness during an emergency situation. The mission-related tasks 
simulated a tactical reconnaissance mission and required use of all of the primary flight symbology 
in contour and NOE flight modes. 

A detailed description of the tasks, conditions, and performance standards follows: 

2.1.2 Hovering Turn (Criterion Maneuver) 

This task was not used as a data collection task. Its purpose was to determine whether the test 
subject pilots had the required skill in handling the aircraft and using the symbology to execute a 
constrained task to acceptable standards. The hover turn (fig. 16) is an ADS-33 task and the criteria 
for adequate performance were the published UCE 2 performance criteria. 

a. Reference : ADS-33, March 2000, par 3.11.17, TC 1-251, ATM Task 1107, Hovering Flight, 
ADS-46 (Draft), June 1998, Par. 7.9.1. 

b. Objectives : This maneuver checks the pilot’s ability to use the display symbology to recover 
from a rapid hovering turn with sufficient precision to fire a weapon. This is an aircraft pointing 
exercise. 

c. Requirements : This maneuver was flown using a precise target heading approximately 180° 
from the initial aircraft heading. 

d. Maneuver : From a stabilized hover at an altitude of not less than 20 ft, complete a rapid 1 80° 
hovering turn to line up with a specified heading while maintaining starting position over the 
ground and altitude. Turns will be completed in both directions. 
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Figure 16. Hovering Turn. 


e. Performance criteria: 

Desired performance : 

• Maintain longitudinal and lateral position within 6 ft of start position. 

• Maintain altitude within ±3 ft. 

• Stabilize final aircraft heading within ±5°. 

• Complete the turn within 15 seconds from the initiation of the maneuver. 
Adequate performance : 

• Maintain longitudinal and lateral position within 12 ft of start position. 

• Maintain altitude within ±6 ft. 

• Stabilize final aircraft heading within ±10°. 

• Complete the turn within 15 seconds from the initiation. 
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2.1.3 Hover Turn Bob-up 

The pilot’s head must be turned approximately 45° in the direction of turn prior to initiating the 
maneuver. The pilot will initiate a 90° hover turn while simultaneously executing a bob-up 
maneuver to 50-ft AGL above the initialization altitude of 10 ft (60-ft AGL) (see fig. 17). 


a. Reference: ADS-46 (Draft), June 1998, Par. 7.9.1, and 7.9.3, ADS-33, March 2000, Par. 3.11.6, 
4.2.2, TC 1-251, ATM Task 1107, Hovering Flight, and Task 1151, Masking and Unmasking. 

b. Objectives: This maneuver checks the pilot’s ability to use the display symbology to control a 
rapid hovering turn coupled with a rapid bob-up maneuver with sufficient precision to fire a 
weapon after completion of the climbing turn. This simulates rising vertically from a masked 
firing position while quickly turning to engage a target 90° off axis. 

c. Requirements: This maneuver will begin on a cardinal heading at approximately 10-ft. AGL. 

The pilot will turn 90° while simultaneously climbing to 60-ft AGL. 

d. Maneuver: This maneuver represents the combining of two different maneuvers specified in the 
references above. Those maneuvers are the turn to target and the hover bob-up. From a stabilized 
hover at an altitude of approximately 10 ft, the pilot will turn his head approximately 45° in the 
direction of the turn. The pilot will complete a 90° hovering turn while simultaneously climbing 
to a hover altitude of 60-ft AGL. The objective will be to arrive at 60-ft AGL at approximately 
the same time the specified heading alignment is achieved. Turns will be completed in both 
directions. The pilot's head may be repositioned as desired after commencing the maneuver. 



Figure 17. Hover Turn Bob-up. 
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e. Performance criteria: 


Desired performance : 

• Maintain longitudinal and lateral position from the starting point within 6 ft. 

• Climb to and maintain 60-ft AGL ±5 ft. 

• Stabilize final aircraft heading ±4° of the target. 

• VSI less than 100 fpm up/down 

• Complete the turn in 15 seconds or less from the initiation. 

Adequate performance : 

• Maintain longitudinal and lateral position of a selected point on the aircraft within 10 ft of a 
selected point on the ground. 

• Climb to and maintain 60-ft AGL ±10 ft. 

• Stabilize final aircraft heading ±6° of the target. 

• VSI less than 200 fpm up/down 

• Complete the turn in 20 seconds or less from the initiation. 

2.1.4 Unusual Attitude Recovery 

Symbology sets designated as primary flight symbology must be capable of assisting the pilot in 
recovering from an unusual attitude caused by inadvertent flight into instrument meteorological 
conditions or spatial disorientation caused by other factors (fig. 18). This is considered an 
emergency maneuver. As such, the simulator evaluation included extreme attitudes, both angle of 
bank coupled with extreme pitch angles. 

a. Reference : ADS-46 (Draft), June 1998, Par. 7.12.1. TC 1-251, Aircrew Training Manual, AH-64 
Attack Helicopter, Task 1245. 

b. Objectives : There are three objectives for this test: 1) The first is to check the adequacy of the 
attitude symbology, 2) Check the utility of the display in allowing the pilot to recognize and 
recover from unusual attitudes, and 3) Check for undesired pilot-vehicle-display dynamics 
during extreme conditions. 

c. Requirements : Four different combinations of pitch and bank entries will be flown. 

d. Maneuver : The aircraft will be initialized in a decelerating attitude at 20 kts and 70-ft AGL. The 
simulator operator will fly the aircraft into a predetermined unusual attitude at which point the 
flight controls will be given to the evaluation pilot to recover the aircraft. The pilot must recover 
to a straight and level attitude with minimal vertical rates using standard UA recovery 
procedures. 
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Figure 18. Low Altitude Unusual Attitude Recovery. 


e. Performance criteria: 

Desired performance : 

• Appropriate initial control input 

• Stabilize wings level ±5° angle of bank 

• Stabilize a constant heading 

• Less than 100 fpm vertical rate 

• Recovery time 15 seconds or less from initiation 

Adequate performance : 

• Appropriate initial control input 

• Stabilize wings level ±10° angle of bank 

• Stabilize a constant heading 

• Less than 200 fpm vertical rate 

• Recovery time 20 seconds or less from initiation 

2.1.5 Tactical Mission 

Execution of this task requires the use of nearly every primary flight symbol presented on the HMD 
while flying a realistic flight profile representative of one of Comanche's primary missions. 

a. Reference: TC 1-251, Aircrew Training Manual, Attack Helicopter, AH-64D, April 1998, Task 
1035, Terrain Flight. 
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b. Objectives: While using the HMD as the primary flight reference, perform contour and NOE 
flight profiles to designated waypoints while maintaining altitudes and airspeeds appropriate to 
the flight mode. The pilot will be searching for ground targets during the NOE portion of this 
mission. 

c. Requirements: AFCS will be used during the maneuver. The maneuvers were flown on pre- 
designated courses in the terrain database. 

2.1.6 Contour Flight Mode 

Contour flight is characterized by constant altitude above ground level (AGL) (generally between 
50-150 feet) and relatively constant airspeed (typically between 60-80 knots) depending on 
vegetation, obstacles, and visual conditions (fig. 19). The flight profile generally follows the contour 
of the surface of the earth. 

The contour flight was initialized over a road. The pilot was required to follow the road that curves 
through rolling terrain using contour flight techniques until arriving at the destination waypoint. 
During the flight, the experimenters commanded two altitude and two airspeed changes. The pilot 
was required to comply as rapidly as possible and to report stable at the new parameter. Time and 
accuracy in achieving and maintaining the new parameter were measured. 


Constant 

Airspeed 

► 


Constant 

Altitude 


Contour Flight Mode 




Figure 19. Contour Flight Mode. 


Desired performance contour flight: 

• Maintain a flight profile that generally conforms to the contour of the earth. 

• Maintain assigned airspeed ±5 kts. 

• Maintain assigned altitude ±10 ft. 

• Maintain aircraft in trim. 

Adequate performance contour flight: 

• Maintain a flight profile that generally conforms to the contours of the earth. 

• Maintain assigned airspeed +10 kts. 
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• Maintain assigned altitude ±15 ft. 

• Maintain aircraft in trim. 

2.1.7 NOE Flight Mode 

NOE flight is characterized as flight at varying altitudes and airspeeds (variable <45 kts) as close to 
the earth's surface as vegetation, obstacles, and visual conditions permit (fig. 20). NOE flight was 
conducted through rolling terrain along a generally prescribed route along four consecutive 
waypoints. While navigating the NOE flightpath, the pilot used visual scanning techniques to 
identify targets (vehicles) to the left and right of the aircraft. The pilot was required to call out the 
azimuth from present position of each vehicle sighted. The time from target sighting to calling out 
the azimuth was recorded along with the difference between the perceived and actual target azimuth. 



ID Targets Wpt3 


Figure 20. NOE Flight Mode. 


d. Performance criteria: 

Desired performance NOE flight: 

• Maintain an altitude that allows safe obstacle clearance while remaining as close to the earth as 
possible while flying a Nap-of-the-Earth profile. 

• Maintain airspeed appropriate for the terrain and visual conditions (generally <60 kts). 

• Use flight symbology to accurately navigate to all waypoints. 

• Accurately call out azimuth to target. 

Adequate performance NOE flight: 

• Maintain an altitude that allows safe obstacle clearance while remaining as close to the earth as 
possible while flying a nap-of-the-earth profile. 

• Maintain airspeed appropriate for the terrain and visual conditions (generally <60 kts). 
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• Use flight symbology to accurately navigate to all waypoints. 

• Accurately call out azimuth to target. 


2.2 Performance Evaluation Tools 


2.2.1 Data Collection 

Several different data collection methods were used to support the simulation as detailed in the 
paragraphs below. 

Show Print and RUNDUM Data Set 

RUNDUM data (e.g., time, position, altitude, heading, other) were collected at 60 Hz, recording the 
variables specified for each maneuver. 

The data collected on the SHOW PRINT file were used as post-run feedback of the pilot’s 
performance following each maneuver. The data were in summary format and included the selected 
performance parameters tailored for each task. 

The parameters for the SHOW PRINT and RUNDUM data are in Appendix B. 

X-Y Plotter 

An X-Y plotter (electronic form) provided researchers with a real-time graphical representation of 
aircraft vertical and horizontal position during the maneuvers. 

2.2.2 Handling Qualities Ratings (HQRs) 

Cooper/Harper HQRs were taken following the completion of each hover bob-up maneuver data set. 
Pilot comments were also recorded that provided rationale for the rating. A copy of the 
Cooper/Harper HQR rating scale can be found in Appendix C. Pilot comments can be examined in 
Appendix G. 

2.2.3 NASA-TLX 

NASA-TLX workload rating scale data were collected immediately following each maneuver data 
set. The scale was filled out based on a subjective assessment made by the pilot concerning mental, 
physical, and temporal demand. This test instrument also captured data concerning how the pilot felt 
about his performance of the task, the level of effort put into the task, and the level of frustration 
experienced in completing the maneuver. A copy of a sample NASA-TLX data collection sheet is in 
Appendix D. 
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SECTION 3. SIMULATION TEST RESULTS 


3.1 Simulation Test Results Overview 

Objective and subjective simulation test results are presented in this section. Objective analysis 
includes actual data collected with respect to the performance parameters set up for each maneuver 
(e.g., time to complete, heading, altitude, and position) at the time the maneuver was completed. 
Pilot performance data concerning each performance parameter were statistically analyzed, and a 
comparison was made between Contact Analog and MIL-STD-1295 compressed symbology. The 
statistical analysis information presented shows averaged performance. More detailed performance 
analyses are presented in Appendix F. 

The subjective analysis section examines the summarized results of the HQRs, NASA-TLX 
workload ratings, situational awareness ratings, and the results of the on-line survey. Pilot comments 
are also summarized in this section with unabridged comments located in Appendix G. In some 
cases, the subjective results pointed out more significant differences in performance than were 
indicated by the objective results. 


3.2 Objective Test Results 

Six pilots completed four evaluative maneuvers. The maneuvers were hover bob-up, unusual 
attitude, contour flight, and NOE flight with target azimuth ID. A 2 x 2 x 4 within-subjects design 
ANOVA (Symbology x Block x Trial) with repeated measures was conducted separately on time, 
attitude, and position data for unusual attitude recovery, hover turn bob-up (position hold), and 
hover turn bob-up maneuvers completed in Comanche Automated Flight Control System (AFCS) 
mode. Those events that produced a statistically significant outcome are presented in the figures 
below. A total objective data set can be found in Appendix F. 

Figure 21 shows that a significant main effect of symbology was found in recorded time to complete 
the hover turn bob-up (Position Hold ON) maneuver, F (1, 5) = 18.03, p < .01, such that 1295 
symbology (M = 13.35) yielded more rapid maneuvering than Contact Analog symbology 
(M = 15.37). No significant differences between symbologies were found in time, attitude, or 
position data in all other maneuvers. 

The results from NOE flights are shown in figures 22 and 23. Figure 22 revealed improved accuracy 
in heading to target estimation when utilizing Contact Analog symbology, F (1, 5) = 12.67, p < .05. 
Likewise, figure 23 shows that shorter response times from target recognition to heading callout 
were recorded utilizing Contact Analog symbology, F (1, 5) = 13.84, p < .05. These results clearly 
show that the head-referenced heading tape associated with the Contact Analog design was faster 
and more accurate in completing this task. 
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Figure 21. Hover Turn Bob-up Time to Complete (Position Hold). 




Figure 22. NOE Mission Performance: 
Accuracy of Heading to Target Estimation. 


Figure 23. NOE Mission Performance: Mean 
time to Call Heading to Target. 
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TABLE 1. OBJECTIVE PERFORMANCE RESULTS SUMMARY 


Maneuver 

Compressed 

Contact Analog 

UA Recovery Time 

Red 

Red 

Bob-up (AFCS) 
Longitudinal Error 

Red 

Red 

Bob-up (AFCS) Lateral 
Error 

Red 

Red 

UA Vertical Rate 

Yellow 

Yellow 

Bob-up (VelStab) Lateral 
Error 

Green 

Yellow 

Bob-up (VelStab) Time to 
Complete 

Green 

Yellow 

Bob-up (AFCS) Time to 
Complete 

Yellow 

Yellow 

NOE Heading to Target: 
Accuracy 


7 X More Accurate 

NOE Heading to Target: 
Speed 


1.3 X Faster 


Green = Desired 
Yellow = Adequate 
Red = Exceeded adequate 



The summary of performance results presented in Table 1 shows that the average recovery time to 
complete the Unusual Attitude task exceeded adequate for both symbology sets. It also shows that 
pilots exceeded adequate standards with regard to holding lateral and longitudinal position with 
AFCS Hover Hold OFF using both symbology sets. Pilots flying both symbology sets only achieved 
adequate performance in maintaining the appropriate vertical rate measured during the UA 
maneuver. They also achieved adequate standards in the time to complete the bob-up with position 
hold OFF. These results clearly show that the decreased performance was experienced using both 
symbology sets with no significant performance difference between the two. The only maneuvers 
where performances differences were noted between symbology sets were the bob-up lateral error 
with VelStab ON, and in bob-up time to complete with VelStab ON. In both instances, performance 
with 1295 was desired whereas performance with Contact Analog was only adequate. The other 
notable difference between symbology sets was that the average pilot was 7 times more accurate and 
1.3 times faster in calling out the heading to target using Contact Analog than when using the 
compressed heading tape on the 1295 symbology presentation. The importance and operational 
significance of this finding was previously discussed. 
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3.3 Subjective Test Results 


Subjective test results are presented and analyzed below. Only those ratings that showed a 
significant finding are presented. Those subjective areas that were analyzed but did not show 
significant findings are presented in Appendix F. Handling qualities ratings (HQRs), NASA TLX 
workload ratings, and situational awareness ratings are presented in this section. 



Compressed AFCS Contact Analog AFCS 


Symbology/Flight Mode 

Figure 24. Bob-up HQRs VelStab Position Hold ON. 


Figure 24 clearly shows a large difference in the difficulty of the maneuver when executing it with 
Position Hold ON (VelStab) versus Position Hold OFF (AFCS). MIL-STD-1295 symbology shows 
an average rating of HQR 3. This translates to “Fair handling qualities with some mildly unpleasant 
deficiencies requiring minimal pilot compensation required for desired performance.” Contact 
Analog symbology was rated approximately HQR 4. This translates to “Minor deficiencies requiring 
considerable pilot compensation to achieve desired performance.” Pilots were able to achieve 
desired standards flying both symbology sets. The difference here was that it required slightly 
greater pilot compensation to fly Contact Analog symbology. 

Flying either symbology set using AFCS (Position Hold OFF) showed an almost doubling of the 
HQRs with higher scores meaning worse handling qualities. Pilots rated Contact Analog and 
compressed 1295 symbology identically with an average rating of 6.54. This means some scored 
HQR 7 and some HQR 6. An HQR of 6 translates to “Very objectionable but tolerable deficiencies 
requiring extensive pilot compensation to attain adequate performance.” An HQR 7 means “Major 
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deficiencies. Adequate performance not obtainable with maximum pilot compensation. 
Controllability not in question.” This clearly shows that without aircraft Position Hold, this was a 
very workload intensive task that required a great deal of pilot compensation to achieve adequate 
performance. This chart clearly shows, however, that there were no differences between symbology 
sets. It is possible that position keeping without position hold may have affected performance so 
severely that no performance differences were found between symbologies. Potential contributors to 
the difficulty of this task were the reduced usable cue environment looking at the outside scene and 
the scaling of the velocity vector and acceleration cue. 

Figure 25 shows that on a scale of 0 to 100, the average overall workload for both compressed and 
Contact Analog symbology was slightly above the middle rating, which is not excessive. It also 
clearly indicates that there was very little difference in the workload between Contact Analog and 
MIL-STD-1295 compressed symbology. This finding was surprising because both verbal and 
written pilot comments led the research team to an expectation of higher workload associated with 
Contact Analog symbology. Figure 26 indicates that the highest workload experienced was for the 
bob-up maneuver followed by the unusual-attitude recovery. The workload was calculated for the 
AFCS flight control mode with Position Hold OFF. There were no significant differences between 
Contact Analog and compressed symbology for any of the individual maneuvers. 


Overall Workload across Symbology Sets 
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Figure 25. NASA-TLX Workload Ratings. 
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TLX Scores for Symbology Sets across Task Types 



□ Compressed 
■ Contact Analog 


Task Types/Maneuvers 


Figure 26. NASA-TLX Scores for All Maneuvers Completed. 


Situational awareness (SA) was defined to the pilots as follows: Knowledge of the interrelationships 
between other aircraft or objects and one’s own position in time and in 3 -dimensional space, such 
that one may operate in the best offensive and defensive manner possible. Pilots were asked to rate 
their situational awareness (SA) for each maneuver flown on a scale from left to right beginning 
with very low to very high. Situational awareness ratings shown in figure 27 indicate there were no 
significant differences between symbology sets. The statistics shown above were the average ratings 
given by all pilots combined and averaged for all tasks flown. SA ratings were taken following the 
completion of each individual maneuver for each symbology and flight control mode. Once again, 
these statistical results seemed inconsistent with the written and verbal comments received from the 
pilots. There were frequent comments related to the lack of situational awareness recorded with the 
HQRs and also during the post-simulation reviews. 


Overall Situation Awareness across Symbology Sets 


< 

CO 


Q) 

> 

o 



□ Situation Awareness 


Symbology Sets 


Figure 27. Averaged Situational Awareness Ratings for All Tasks. 
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Figure 28 shows the SA ratings for the individual tasks flown. This detailed analysis shows no 
significant differences between symbology sets when looking at the individual maneuvers. SA was 
better for the hover bob-up task and worse for the terrain flight tasks. This was a predictable result 
given the reduced visual cues provided by the FLIR image while executing near-earth flight 
maneuvers in rolling and wooded terrain. 


Situation Awareness for Symbology Sets across Task Types 
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Figure 28. Situational Awareness Ratings for Individual Tasks. 


A predicted result that was not yielded was the SA ratings for the unusual-attitude maneuver. The 
differences in the horizon line design philosophy between the symbology sets was expected to show 
a contrast during this maneuver. The Contact Analog design fixes the horizon line to the true 
horizon, whereas the MIL-STD-1295 horizon line is screen-fixed. The Contact Analog design 
caused the horizon line to disappear from view during nose-up attitudes, leaving no visual horizon 
reference except the blank sky. In spite of this fact, pilots were able to level the aircraft quickly 
enough to manage this deficiency. The 1295 screen-fixed horizon line and iron wings were in view 
at all times during the maneuver. Although all pilots commented on the lack of a Contact Analog 
horizon reference during the UA, they did not seem to reflect those comments in the SA ratings. 

Figure 29 shows the averaged handling-qualities ratings for the bob-up maneuver flying three hybrid 
symbology sets using two different flight control modes. The only VelStab selectable mode engaged 
for this maneuver was Position Hold ON. The primary difference between the two test conditions 
was that the hover position over the ground was automatically held by VelStab for one condition, 
whereas the position was held by pilot input for the AFCS condition. 
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Figure 29. Hybrid Symbology HQRs - Bob-up. 


The difference in workload between the two flight control modes was evident in the average HQRs 
reflected in the data. The data show a very high workload in the hand-flown AFCS mode. This likely 
confounded any distinguishable differences between the three hybrid symbology sets. In the VelStab 
Position Hold mode, an HQR of 4 shown for the CA roll-fixed and 1295 replaced hybrids is charac- 
terized by “Minor deficiencies. Desired performance required considerable pilot compensation.” 
HQR 5 represented by the 6-sec predictor translates to “Moderately objectionable deficiencies. 
Adequate performance requires considerable pilot compensation.” The major difference between 
HQR 4 and 5 is that HQR 5 is the breakpoint for moving from a desirable performance rating to an 
adequate rating. An average HQR of 6.5 was scored for all three hybrid symbology sets flown in the 
AFCS mode (between HQR 6 and HQR 7, there is no formal HQR 6.5). An HQR 6 is defined as 
“Very objectionable but tolerable deficiencies. Adequate performance requires extensive pilot 
compensation.” An HQR 7 means “Major deficiencies. Adequate performance not obtainable with 
maximum pilot compensation. Controllability not in question.” The 6.5 rating means pilots were 
only able to achieve adequate performance about half the time but the aircraft was never out of 
control. 

The drift deviation from the starting hover position in the AFCS mode often exceeded 45 feet 
laterally or longitudinally and sometimes both. The amount of drift was not predictable and difficult 
to control given the scaling of the hover symbology. For this simulation there was no “home plate” 
symbol that pilots could use to maintain position. That symbol was subsequently implemented in the 
Sikorsky Engineering Development Simulator. It is unknown whether that symbol has been tested. 
The operational significance of the excessive amount of drift recorded in the AFCS is unknown. 

To summarize, in a comparison of means calculated from NASA-TLX ratings, collapsed across 
subscales, no significant difference was found as an effect of symbology condition. Additionally, 
aircraft handling quality ratings (HQRs) did not differ as a result of symbology condition for the 
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hover turn bob-up maneuvers. Ratings on both NASA-TLX and HQRs revealed no differences in 
perceived workload, situational awareness, or aircraft handling qualities for compressed and Contact 
Analog symbologies. 


3.4 Online Questionnaire Results 

Pilots' subjective ratings on the effectiveness of the display symbology were collected in an online 
questionnaire following testing within a symbology set. Selected individual display symbols (i.e., 
horizon line reference) were rated individually to determine usefulness in completing the task and 
ease or difficulty in controlling related aircraft axes of motion. 

The following items were rated utilizing the scale found in Appendix E. Ratings were taken in an 
online format with mouse input on a slider scale with endpoints corresponding to those listed. 
Significant findings of variance as a result of symbology have been graphed. Only those ratings that 
produced a significant finding are presented. 


3.4.1 Legend for All Graphs 
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Figure 30. Average Rating of Correct Symbol Position within a Symbology Set. 


Figure 30 shows that pilots rated the position of the symbols in the 1295 compressed symbology set 
to be nearly correct (90%) for all symbols. The rating for Contact Analog was only about 65%, 
meaning that nearly a third of the symbols were not considered to be correctly positioned. Pilots 
commented that the Contact Analog symbology set was too widespread. The position of the symbols 
was pushed toward the outer edges of the usable HMD display area. The design rationale was 
reportedly to declutter the central viewing area of the display to allow the pilot to better see targets 
or objects of interest without symbology interference. Pilots reported that positioning the symbols 
outward slowed their crosscheck and increased workload. The 1295 symbology was positioned 
much closer to the central viewing area of the display and therefore received a more favorable 
rating. 
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Figure 31. Horizon Reference Line Readability for Each Symbology Set. 


Pilots’ satisfaction with the readability of the 1295 horizon line was approximately 85% compared 
with only 48% for Contact Analog (fig. 31). The 1295 horizon line was screen-fixed. The 
relationship between the horizon line and the iron wings was constant regardless of the pilot’s head 
position. The Contact Analog horizon line was fixed on the true horizon and allowed the gap in the 
center of the line to move with the pilot’s head. The iron wings aircraft referenced them viewable 
only when looking straight forward over the nose of the aircraft (much like a virtual HUD). The 
earth-referenced horizon line design feature caused the horizon line to remain fixed on the true 
horizon during up and down head movement and aircraft pitch movement. The horizon line was 
therefore not visible or usable during portions of maneuvers such as an unusual attitude, steep 
descents, or other maneuvers where the aircraft or pilot’s head was up or down. This directly 
affected the Contact Analog readability ratings. 
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Figure 32. Ease with Which Pilots Could Control Aircraft Pitch. 


Pilots rated control of aircraft pitch at approximately 78% for the 1295 symbology set compared 
with only approximately 50% for Contact Analog (fig. 32). Comments in the above paragraph with 
reference to the horizon line are likely reflected in this rating. As stated previously, the 1295 horizon 
line was screen-fixed and appeared as though painted on the pilot’s visor, moving with head 
movement and never-changing position in the central field of view. The position of the iron wings 
and the horizon line could be seen no matter where the pilot was looking (i.e., up, down, left or 
right). With Contact Analog, the iron wings were aircraft-referenced, meaning they remained 
positioned over the nose of the aircraft and could only be viewed by the pilot when looking 
relatively straight forward. When looking off-axis during turns (a maneuver requirement) the iron 
wings could not be seen and the pilot had no pitch reference symbol for use in controlling aircraft 
attitude. In addition, the gap in the Contact Analog was much wider than the width of the iron wings. 
Therefore, there was no way to precisely match the level of the wings with the horizon line. Pilots 
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could not superimpose the iron wings over the horizon line to achieve a precise level attitude, as is 
common practice with more traditional attitude reference symbology. 



0 20 40 60 80 100 


Unsatisfactory < — > Satisfactory 


Figure 33. Vertical Situation Indicator Readability for Each Symbology Set. 


Pilots preferred the VSI readability of the 1295 symbology (fig. 33) over Contact Analog at a rate of 
90% satisfied compared with approximately 55% satisfied for Contact Analog. Based on pilot 
comments, the most probable reason was the lack of a scale in the Contact Analog VSI that would 
provide an analog cue for determining rates. The length of the VSI line was the intended analog cue 
for determining rate of climb information with a digital readout at the bottom of the scale. Pilots 
found the length of the line to be a less compelling rate cue than the pointer against a fixed scale 
used in the 1295 symbology design. 



Figure 34. Dynamics of the Vertical Situation Indicator Symbol. 


Satisfaction with the dynamics of the 1295 vertical rate symbology (fig. 34) was rated nearly twice 
that of Contact Analog. The primary reason given by pilots was that flying the pointer against a 
fixed vertical velocity scale provided a more predictable result when collective was increased or 
decreased during the hover bob-up or when holding an altitude during a rapid hovering turn. The 
Contact Analog VSI symbology had no scaling or index. It did present a digital readout. The 
beneficial effect of having a compelling analog rate indication was missing from the Contact Analog 
VSI symbology. This can also be seen in figure 35. 
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Figure 35. Control of Aircraft Rate of Climb and Descent. 


Figures 33, 34, and 35 relate to the readability, dynamics, and ease of controlling vertical rates using 
both symbology sets. Ease of controlling vertical rates was rated much easier for 1295 symbology 
(75% easy) than for Contact Analog, which was weighted at approximately 42%. The 1295 design 
for the Vertical Speed Indicator (VSI) was a traditional moving pointer against a fixed scale that was 
graduated in 100 feet per minute (fpm) rate of climb or descent for the first 500 fpm and 500-fpm 
increments above that amount. The Contact Analog scale was a vertical line that grew longer with 
increased vertical rates and retracted to be invisible when rates were null. There was not a pointer or 
graduated scale associated with the line. However, there was a digital readout. Pilot comments 
reflected that maintaining a constant altitude or maintaining a constant rate of climb or descent was 
easier and required less workload using the pointer against the scale than using the Contact Analog 
line only with digital readout design. Pilots reported they could be more aggressive and precise in 
maintaining altitude in a level hover turn using the 1295 design. It was reportedly easier to null 
vertical rates using the pointer without getting into pilot-induced oscillation (PIO) associated with 
leveling out during the bob-up maneuver. 
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Figure 36. Satisfaction with the Dynamics of the Radar Altimeter Symbol. 


Pilots reported a 95% satisfaction rate with the 1295 radar altimeter design compared to only a 63% 
satisfaction with the Contact Analog design (fig. 36). The 1295 radar altimeter design is a vertical 
thermometer against a fixed scale with various graduated heights in feet varying from 10-ft to 100-ft 
increments depending on altitude above the ground. It also provides a digital readout of altitude 
AGL in one-foot increments. The lower the altitude the more detailed the graduations. The Contact 
Analog design is a vertical thermometer type design with the first graduated scale at 100 feet, and 
two others at 500-ft increments separated by a non-linear distance between the graduations. There is 
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a digital readout at the base of the symbol graduated in one-foot increments. Pilots commented that 
the non-linear scaling of the Contact Analog design was misleading with regard to determining rates 
using the thermometer scale. They also commented on the need to have a scale in smaller increments 
for altitudes below 100-ft AGL for use in maintaining ground clearance during IGE hover 
maneuvers. With Contact Analog, pilots tended to use the digital readout alone as the primary source 
of rate and height information and the thermometer scale was not used. 



Figure 37. Use of Symbology for Accurately and Rapidly Reporting Heading to Target Information. 


Pilots overwhelmingly preferred Contact Analog symbology over 1295 for rapidly reporting heading 
to target by a margin of almost two to one (fig. 37). A design feature of the head-tracked Contact 
Analog heading tape is that the heading tape can overlay a target through head movement and the 
target bearing can be quickly and accurately determined. Using the screen-fixed 1295 symbology, 
the pilot had to estimate how far the target was left or right of the aircraft centerline, and add or 
subtract that amount from the current aircraft heading. This took more time and was less accurate 
than using the head-tracked heading tape. 


3.5 Summary of Subjective Questionnaire Ratings 

Pilot ratings revealed major differences as a result of symbology type on 8 of 35 questions (fig. 38). 
Subjective ratings revealed the perceived benefit of 1295 compressed symbology for 7 of those 8 
questions. Performance benefits included higher ratings of effectiveness for display items and 
greater ability to control related axes when utilizing 1295 compressed symbology. Specifically, 
pilots gave higher satisfaction ratings for readability of the horizon reference line (M = 84% 
satisfaction) and vertical speed indicator (M = 90% satisfaction). Furthermore, pilots viewed related 
dynamics of pitch (M = 78%) and rate of climb/descent (M = 75%) as more easily controlled when 
flying with 1295 symbology. Correct placement of symbols within 1295 symbology (M = 90% 
correct) was also rated higher than Contact Analog symbology (M = 65% correct). Lastly, pilots 
reported higher levels of satisfaction with the dynamics of the radar altimeter symbol in 1295 
symbology (M = 95% satisfaction) than Contact Analog symbology ( M = 65% satisfaction). 

In a reversal of preference, pilots rated Contact Analog symbology more useful ( M = 76%) than 
1295 symbology ( M = 28%) for accurately reporting heading to target information. This finding was 
not surprising considering the Contact Analog design philosophy that supports a veridical overlay of 
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heading information analogous to its world counterpart. It should be noted that pilots received 
feedback on the accuracy of their target heading reports prior to completing the online questionnaire. 

In summary, pilots significantly favored 1295 symbology for the readability, placement, and 
dynamics of the symbols in flight maneuvers. Pilots judged Contact Analog symbology to be more 
useful than 1295 symbology for reporting target headings in an operational scenario. 


Control a/c pitch 
Correct symbol pos. 
Horiz. line readability 
Rad alt symbol dynamics 
Rate of climb/desc. 
Reporting target hdg 
VSI readability 
VSI symbol dynamics 



Average Subjective Rating (%) 


Figure 38. Summary of Online Questionnaire Ratings. 


3.5.1 Summary of Pilot Comments 

Pilot comments were recorded following the HQR ratings for the hover bob-up tasks after the 
completion of each maneuver for each symbology set. The total set of comments can be reviewed in 
Appendix G. Additional end-of-simulation summary comments were also recorded. Test team 
members reviewed all comments provided by the pilots in an attempt to find trends and common 
issues with respect to the symbology sets flown during the test. The following is a summary of the 
most repeated comments recorded by the test team. 

3.5.2 Heading Tape 

Six of six pilots stated that the Contact Analog heading tape moves too fast during rapid head 
movement or rapid aircraft yaw or turns to be usable as an analog cue. The heading tape blurs and is 
unreadable during turns. Pilots commented they had to constantly turn their head back over the nose 
to see the heading index. 

Six of six pilots commented they could not differentiate between head movement and aircraft 
movement during turns or rapid yaw maneuvers when looking off axis. Combining heading tape 
movement with head movement gives the feeling that aircraft yaw rates are greater than actual rates. 
The ability to see the target heading during the majority of the turn using a compressed heading tape 
(1295) reduced workload and increased crosscheck time available for other parameters. 
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Pilots questioned the operational value of head-referencing the heading tape. Pilots stated there are 
other ways to get this same information without head-referencing the entire heading tape. Apache, 
for example, provides for an alternate crewmember HMD LOS bearing and a copilot/gunner (CPG) 
HMD LOS symbol, both located along the bottom of the compressed heading tape, to indicate the 
exact azimuth to a target that the pilot or CPG is looking at. 

3.5.3 Horizon Line 

Six of six pilots commented that the Contact Analog horizon line disappears from view during 
aircraft nose up/down or head up/down conditions, leaving the pilot with no horizon reference. Not 
having a visible horizon could present an unsafe situation. 

Five of six pilots commented that the earth-referenced horizon line was difficult to use with respect 
to unusual attitudes and was often not present in the field of view. This can result in significant 
disorientation especially during the initial phases of an unusual-attitude recovery. 

There were additional comments concerning the width of the gap in the middle of the horizon line. 
Recommendations were to reduce the width of the gap from 18° to something less where the tips of 
the iron wings would touch the horizon line in a level attitude. The design rationale for the 18° gap 
was reportedly to reduce clutter in the central viewing area of the HMD display. The comments on 
this subject indicated it was a lower priority issue and not flight critical. 

3.5.4 Six-Second Predictor 

Six of six pilots commented that the six-second predictor was not usable in stopping climb/descent 
at a desired radar altitude. 

Six of six pilots commented that attempts to use the six-second predictor resulted in pilot- induced 
oscillations and longer times to stabilize at a desired altitude. 

3.5.5 Symbology Position 

All six pilots commented that the Contact Analog symbology was generally positioned too far to the 
outside edges of the display. They commented that the widespread symbology design increased 
workload, slowed crosscheck, and decreased readability. This reportedly resulted in longer 
maneuver completion times. 

3.5.6 Torque Symbol 

All six pilots commented that the torque symbol is positioned closer to the central viewing area of 
the display than more important symbology, such as the radar and barometric altitude symbology. 
Pilots commented that altitude information is more critical for situational awareness and survival 
than torque. Pilots used the digital portion of the torque symbology but felt the thermometer scale 
was of no use in executing the test maneuvers. 

3.5.7 Hover Position Cue 

All six pilots commented on the need for hover position symbology to determine the control inputs 
required for maintaining hover position. A hover position box was not provided in the Contact 
Analog symbology set used for this simulation. This symbol was specified in the PYIMS but at the 
time of the Ames simulation, it had not been implemented or tested by Sikorsky. 
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3.6 Validation of Test Results 


The Technical Division Chief for PM Comanche directed that a single, coordinated position be 
established between the participating pilots and the test team members concerning the findings and 
recommendations of the HMD symbology test. This meant agreeing upon the meaning of the 
objective and subjective test findings and establishing a coordinated position concerning 
recommendations for improvements and the priority for addressing the recommendations. The 
coordinated results were briefed to the Comanche PM, and subsequently to the Sikorsky Chief 
Engineer for Comanche along with the Comanche Crew Station Working Group (CSWG) made up 
of both Government and Sikorsky technical personnel. 

The test team carefully reviewed the objective and subjective test results to identify specific trends 
in preparation for briefing the participating pilots. This review looked for repeated pilot difficulties 
with certain tasks, repeated HQR comments, common subjective questionnaire ratings, and end-of- 
simulation interview comments. The team identified areas where a clear consensus was achieved by 
all or most of the six pilots concerning the symbology flown during the simulation. This review 
identified six Contact Analog symbols that warranted change. These were the heading tape, horizon 
line, overall symbology positioning in the FOV, the six-second radar altimeter predictor, the torque 
symbol, and the lack of hover position cue. 

A prioritization rating scheme was developed that required the pilots to rate the seriousness of the 
perceived deficiencies in each of the six symbols. The prioritization scheme follows: 

(1) Must change now. Safety of flight, workload, or performance issue that must be addressed prior 
to flight. 

(2) Should be changed at the earliest opportunity. Violates best design practices, human factors 
principles. Should be changed in next block upgrade. 

(3) Nice to have. If possible, change at earliest convenience when making other modifications. 

The test results were then briefed to the participating ATTC pilots and Ames pilots. Using the above 
rating scheme, the pilots arrived at a consensus opinion that of the six symbols identified as being 
problematic, three were considered to be a priority one for improvement. The priority one symbols 
were the heading tape, horizon line, and six-second predictor. Safety of flight concerns was the 
primary reason given for the priority-one recommendation. The ATTC pilots were unanimous in 
their recommendations, whereas the Ames pilots were mixed between priority two and three. The 
detailed results (figs. 39, 40, and 41) were as follows: 
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3.6.1 Heading Tape 

Comments: Moves too fast to be used as an analog heading cue. 

Tape movement cannot be differentiated between head and aircraft movement. 


Priority Ratings: 1, 1, 1, 1, 2, 2 



Figure 39. Heading Tape Modification Priority Ratings ( n = 6). 


3.6.2 Horizon Line 

Comments: Disappears from field of view in nose up/down or head up/down conditions. 
Priority Ratings: 1, 1, 1, 1, 2, 3 



Figure 40. Horizon Line Modification Priority Ratings (n = 6). 
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3.6.3 Six-Second Predictor 

Comments: Existing predictor design is not usable for precisely stopping a vertical rate at a 
desired altitude. Predictor causes pilot-induced oscillations (PIO). 


*Note: Unanimous consensus on removal of six-second predictor if no further design modifications 

These findings were included in a comprehensive briefing that was presented to both the Comanche 
PMO, the Sikorsky Chief Engineer for Comanche, and to the CSWG on October 22-23, 2003. 


The test team came to the following conclusions after taking all objective and subjective data into 
account. These were briefed to the Comanche PMO and to the CSWG during the October 2003 
briefings referenced in the above paragraph. 

The simulation results show no objective test data that would warrant restricting experimental test 
pilots from flying constrained tasks. 

ATTC safety-of-flight issues warrant further testing and modification of the design of selected 
symbols in the Contact Analog symbology set. The symbols which were rated priority 1, safety-of- 
flight were: 

Heading Tape 
Horizon Line 
Six-Second Predictor 


Priority to Remove: 1, 1, 1, 1, 1, 1 
Priority to Modify: 2, 2, 2, 2, 2, 3 






Figure 41. Six-second Predictor Modification* Priority Ratings ( n = 6). 


3.7 AFDD Test Team Summary and Conclusions 
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Small but consistent advantages were recorded for the MIL-STD-1295 symbology design when 
executing ADS-33 constrained tasks in both this simulation as well as the previous Comanche 
simulation. 

The azimuth-to-target task favored the Contact Analog heading tape design. No other performance 
differences were recorded for the operational maneuvers (NOE, contour flight modes). 

There were consistent and strong pilot comments supporting the MIL-STD-1295 design over 
Contact Analog in both this current simulation and in Comanche Sim I. 

Sikorsky’s Crew Station Design Group has put together a plan for modifying and testing a 
redesigned Contact Analog symbology set. This plan was briefed at the CSWG meeting in January 
2004. 
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APPENDIX A. SYMBOLOGY AND TEST MANEUVER TRAINING 


Each evaluation pilot will be provided training to familiarize the pilot with the simulation cab, flight 
controls, and HMD symbology. The training sessions for symbology familiarization will follow the 
schedule below. Progress through the training sessions will be monitored to determine when the 
pilot is ready to make data evaluations. The training is estimated to take 8-10 hours as follows: 

1-hour classroom Contact Analog Symbology 

1-hour classroom Compressed Symbology 

. 5-hour classroom Hybrid Symbology 

1-2 hours simulator Contact Analog Symbology - Fam + Training Tasks* 

1-2 hours simulator Compressed Symbology - Fam + Training Tasks* 

1-2 hours simulator Hybrid Symbology - Fam + Training Tasks* 

.5-hour simulator Contact Analog Symbology - Training Evaluation f 

. 5-hour simulator Compressed Symbology - Training Evaluation f 

. 5-hour simulator Hybrid Symbology - Training Evaluation t 

Notes: 

The order of training will be counterbalanced. 

This training will be somewhat self-paced and some pilots may not require the full number of hours 
although they will be available. 


* The familiarization flight will consist of structured instruction and free flight. This will be followed by practice on 
specific maneuvers similar to those that will be used in the experiment, 
f The evaluation flight will consist of re-familiarization, and then a criterion task maneuver (hover turn) will be flown to 
the performance parameters defined by ADS-33. The pilot will have 3 attempts at the maneuver. Successful 
completion of the maneuver to standards will qualify the pilot to proceed to data evaluation. If a pilot cannot proceed, 
the project pilot will provide appropriate additional training prior to re-test. 
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APPENDIX B. DATA REQUIREMENTS FOR COMANCHE SIMULATION II 


Experimenter & Pilot Control Requirements 
Post-Run Data File Requirements 
Post-Run Printout Requirements 
Raw Data File Requirements 

For purposes of data recording and processing, there are six maneuvers listed in table 2: 


TABLE 2. MANEUVERS 


A. ) Hovering Turn 

B. ) Vertical Maneuver (note that this maneuver is deleted) 

C. ) Hover Turn Bob-up 

D. ) Unusual Attitude Recovery 

E. ) Contour Flight 

F. ) Nap-of-the-Earth (NOE) 


In the test plan, the contour flight and NOE are listed together as "tactical mission." A different raw 
and processed data file will be generated for each of the six maneuvers. 

Most of the maneuvers have two or more possible starting positions, starting aircraft state, and/or 
desired direction of turn as listed in table 3: 


TABLE 3. MANEUVERS WITH INITIAL CONDITIONS 


Al.) Hovering Turn, Left 

A2.) Hovering Turn, Right (same initial conditions as Al) 


Bl.) Vertical Maneuver (note that this maneuver has been deleted) 


Cl.) Hover Turn Bob-up, Left 

C2.) Hover Turn Bob-up, Right (same initial conditions as Cl) 


Dl.) Unusual Attitude Recovery, Up, Right 
D2.) Unusual Attitude Recovery, Up, Left 
D3.) Unusual Attitude Recovery, Down, Right 
D4.) Unusual Attitude Recovery, Down, Left 


El.) Contour Flight, Course #1, Forward 
E2.) Contour Flight, Course #1, Reverse 
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TABLE 3. CONTINUED 


E3.) Contour Flight, Course #2, Forward 
E4.) Contour Flight, Course #2, Reverse 
E5.) Contour Flight, Course #3, Forward 
E6.) Contour Flight, Course #3, Reverse 


FI.) NOE Flight, Course #1, Forward 
F2.) NOE Flight, Course #1, Reverse 
F3.) NOE Flight, Course #2, Forward 
F4.) NOE Flight, Course #2, Reverse 
F5.) NOE Flight, Course #3, Forward 
F6.) NOE Flight, Course #3, Reverse 


Each type of maneuver (such as the four unusual-attitude recovery maneuvers Dl, D2, D3, and D4) 
will generate the same raw data file, post-run (processed) data file, and printout. However, the initial 
conditions of the data run may be different. 

The file-naming convention is listed in table 4: 


TABLE 4. FILE NAMING CONVENTION 


Raw data files shall have a .rnd extention. Processed data files shall have a .dat or .xls 
extention. 


The experimenter has five toggle buttons and a momentary event marker as listed in table 5. The 
toggle buttons should light up to distinguish one state from the other. Status on a monitor is 
acceptable if lighting the button is difficult. 


TABLE 5. EXPERIMENTER BUTTONS 


1. ) Toggle: Data Recording ON/OFF 

2. ) Toggle: Command Altitude High START/COMPLETE 

3. ) Toggle: Command Altitude Low START/COMPLETE 

4. ) Toggle: Command Airspeed START/COMPLETE 

5. ) Toggle: Command Airspeed Low START/COMPLETE 

6. ) Momentary: Target Provided Event Marker 


The pilot has one momentary switch event marker as listed in table 6. In addition the pilot has an IC 
button to stop the simulation and go back to (paused) initial conditions. 


51 





TABLE 6. PILOT BUTTONS 


1. ) Momentary: Target Seen Event Marker 

2. ) IC button 


Each raw and processed data file consists of ASCII characters, (comma?) separated columns, (line- 
feed?) separated rows. The raw data files will have all information necessary for post-run 
reconstruction of the maneuver, including the 3D position, 3D linear velocities, and 3D orientation 
of the aircraft during the run. In addition, the raw data files must have the head orientation recorded 
(head position not required). The first row of the raw data file contains the text headers labeling each 
column of data. All data in the raw data file will be sampled at the same rate. Each row is a new 
sample. 

Each post-run data file will have at least two rows. The first row consists of text headers labeling. 
The second row and subsequent rows have all the statistical data detailed later in this document. 
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Dependent Measures for Comanche Sim II 
Post-Run Data File and Printout 


Note: Raw data file requirements are not yet listed in this document. 


1. Hovering Turn (Criterion Task Only) 

Post-run data file and printout (showprint): 

Data recording starts when the experimenter's Data Recording button goes into the ON state. When 
the button transitions to the OFF state, the raw data file is saved, the post-run data file is generated 
and saved, and the post-run statistics printout is generated. The post-run data file and printout 
contain the following: 

1) Run number 

2) Date/time 

3) Pilot number (ID) 

4) Task and turn direction (i.e.. Hovering Turn Left) 

5) Longitudinal RMS* [min., max., RMS, performance label (desired, adequate, not adequate)] 

6) Lateral RMS* [min., max., RMS, performance label (desired, adequate, not adequate)] 

7) Altitude RMS [min., max., RMS, performance label (desired, adequate, not adequate)] 

8) Final Heading Delta [performance label (desired, adequate, not adequate)] 

9) Time to Complete [performance label (desired, adequate, not adequate)] — the length of time 
the Data Recording button is in the ON state. 

*Calculated from pilot's seat position (or FLIR sensor?), not CG 

The hovering turn printout is shown in table 7. 


TABLE 7. HOVERING TURN PRINTOUT 


Pilot ID 


Task/tum 

direction 

Run number 

Date 

Time 

Statistics 

Value 

Min. 

Max. 

RMS 

Performance 

Label 

Longitudinal 


X 

X 

X 

X 

Lateral 


X 

X 

X 

X 

Altitude 


X 

X 

X 

X 

Final Heading 

X 




X 

Time to 
complete 

X 




X 
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2. Hover Turn Bob-up 


Data recording starts when the experimenter's Data Recording button goes into the ON state. When 
the button transitions to the OFF state, the raw data file is saved, the post-run data file is generated 
and saved, and the post-run statistics printout is generated. The post-run data file and printout 
contain the following: 

Post-run data file and printout tshowprint): 

1) Run number 

2) Date/time 

3) Pilot number (ID) 

4) Task and turn direction (i.e., hover turn bob-up left) 

5) Longitudinal RMS [min., max., RMS, performance label (desired, adequate, not adequate)] 

6) Lateral RMS [min., max., RMS, performance label (desired, adequate, not adequate)] 

7) Max. Altitude 

8) Final Altitude 

9) Final Heading 

10) Time to Complete [performance label (desired, adequate, not adequate)] — the time the Data 
Recording button is in the ON state 

The hover bob-up printout is shown in table 8. 

TABLE 8. HOVER BOB-UP PRINTOUT 


Pilot ID 


Task/turn 

direction 

Run number 

Date 

Time 

Statistics 

Value 

Min. 

Max. 

RMS 

Performance 

Label 

Longitudinal 


X 

X 

X 

X 

Lateral 


X 

X 

X 

X 

Altitude 



X 



Final Altitude 

X 




X 

Final Heading 

X 




X 

Time to 
complete 

X 




X 
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3. Unusual Attitude Recovery 

Data recording starts when the experimenter's Data Recording button goes into the ON state. When 
the button transitions to the OFF state, the raw data file is saved, the post-run data file is generated 
and saved, and the post-run statistics printout is generated. The post-run data file and printout 
contain the following: 

Post-run data file and printout tshowprint): 

1) Run number 

2) Date/time 

3) Pilot number (ID) 

4) Task and Initial Position (i.e., Unusual Attitude Down Left) 

5) Roll Angle 

6) Pitch Angle 

7) Pitch Rate? 

8) Roll Rate? 

9) Yaw Rate? 

10) VSI vertical rate 

11) Recovery time [performance label (desired, adequate, not adequate)] - the time the Data 
Recording switch is in the ON state 

12) Number of ground strikes 

Append the data in table 9 to the processed data file and printout for the state of the aircraft at the 
beginning of the run, and at the end of the run. 
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TABLE 9. UNUSUAL ATTITUDE RECOVERY PRINTOUT 


Pilot ID 


Task/Initial 

Position 

Run number 

Date 

Time 

Statistics 

Value 




Performance 

Label 

No. of 
ground 
strikes 

X 





Final Roll 
Angle 

X 




X 

Pitch Angle 

X 




X 

Pitch Rate? 

X 





Roll Rate? 

X 





Yaw Rate? 

X 





VSI vertical 
rate 

X 




X 

Recovery 

time 

X 




X 


4. Contour Flight (One of two elements of the Tactical Mission) 


A = Stable at (High/Low) Altitude, (High/Low) Airspeed 
B = Commanded (High/Low) (Altitude/ Airspeed) 

C= 5 Sec after Commanded (High/Low) (Airspeed/Altitude) 



"Operate" Command Command Command 
Altitude Altitude Airspeed 

High High Low 

"Start" "Complete" "Start" 


Command Command Command Command Command 

Airspeed Altitude Altitude Airspeed Airspeed 

Low Low Low High High 

"Complete" "Start" "Complete" "Start" "Complete" 


Append 

Data 

A 


Append 

Data 

B&C 


Append Append 

Data Data 

A B&C 


Append Append 

Data Data 

A B&C 


Append Append 

Data Data 

A B&C 


Append 

Data 

A 


Figure 42. Data Recording Timing for Contour. 
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Post-run data file and printout (showprint): 

Altitude high/low event marker 
Airspeed high/low event marker 
Altitude RMS 
Airspeed RMS 

Final Altitude Delta for Altitude event 
Final Airspeed Delta for Airspeed event 

Time to complete each "event" (experimenter button press marks end of event - delta between initial 
to final button press) 

Pilot stick positions 

There are five experimenter buttons used for this maneuver: 

1. ) Data Recording ON/OFF 

2. ) Command Altitude High START/COMPLETE 

3. ) Command Altitude Low START/COMPLETE 

4. ) Command Airspeed High START/COMPLETE 

5. ) Command Airspeed Low START/COMPLETE 


The raw data file is recording data continuously from Data Recording ON to Data Recording OFF. 

Each time a command button is pressed (either state), or the Data Recording button goes to the OFF 
state, data are appended to the post-run data file and printout. 

Only one of the four command buttons may be ON at a time. Ignore any additional ON presses until 
the four command buttons are all OFF. When data are appended to the post-run file or printout, the 
data shall be as shown in table 10: 

TABLE 10. CONTOUR FLIGHT PRINTOUT 


Pilot ID 

Course 

Latency 

Run number 

Date 

Time 

Statistics 

Value 

Min. 

Max. 

RMSE*** 

Performance 

Label 

State of command * 

X 





Altitude 


X 

X 

X 


Airspeed 


X 

X 

X 


Cyclic Long. 


X 

X 



Cyclic Lat. 


X 

X 



Collective 


X 

X 



Time to complete ** 

X 






*State of the command is the text labeled A, B, or C. Please insert the correct text (High or Low, Altitude or Airspeed). 
**Where Time to Complete is the time from a Command Button START (1 of 4) to the same button COMPLETE 
(Time B). Time to Complete is not recorded or printed for any of the “Stable” conditions listed previously (Time A). 
***RMSE is referenced to the desired radar altitude or airspeed. The VMS software must keep track of what the current 
desired altitude and airspeed is, based on the last buttons pressed for altitude and airspeed (or initial conditions). 
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5. NOE Flight (One of Two Elements of the Tactical Mission) 

Post-run data file and printout (showprint): 

Altitude 
Airspeed 
Target number 
Heading to current target 

Time between a target pop-up and the pilot's event marker 

Time between the pilot's event marker and the experimenter's event marker 

Number of ground strikes 

Number of tree strikes 

Flight time above 50 ft. 

Total time of run 

There are three buttons used for this maneuver: 

1. ) Experimenter's Data Recording ON/OFF 

2. ) Experimenter's "Target Provided" Event Marker 

3. ) Pilot's "Target Seen" Event Marker 


The raw data file is recording data continuously from Data Recording ON to Data Recording OFF. 

Every time the experimenter's event button is pressed, the following statistics (table 1 1) are 
appended to the post-run file and printout: 

TABLE 11. NAP-OF-THE-EARTH DATA APPENDED TO PRINTOUT DURING RUN 


Pilot ID 

Course 

Latency 

Run number 

Date 

Time 

Statistics 

Value 





Target No. 

X 





Heading to current 
target at pilot event 
button press 

X 





Aircraft heading at 
pilot event button 

X 





Time between 
target pop-up and 
pilot event button 

X 

Or 

"never" 





Time between 
pilot event button 
press and 

experimenter event 
button press 

X 

Or 

"never" 
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At the end of the run (when the experimenter changes the state of the Data Recording button to OFF) 
the following statistics (table 12) are appended to the post-run file and printout: 


TABLE 12. NAP-OF-THE-EARTH DATA APPENDED TO PRINTOUT AT END OF RUN. 



Value 

Min 

Max 

RMS 

Mean 

Altitude 


X 

X 

X 

X 

Airspeed 


X 

X 

X 

X 

No. of ground 
strikes 

X 





No. of tree 
strikes 

X 





Flight Time 
above 50 ft. 

X 





Time to 
complete run 

X 
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APPENDIX C. COOPER/HARPER HANDLING QUALITIES RATING SCALES 


HANDLING QUALITIES RATING SCALE 



AIRCRAFT 

CHARACTERISTICS 


DEMANDS ON THE PILOT IN SELECTED 
TASK OR REQUIRED OPERATION* 


PILOT 

RATING 


ADEQUACY FOR SELECTED TASK 
OR REQUIRED OPERATION* 


Minor but annoying 
deficiencies 


Desired performance requires moderate 
pilot compensation 


Is it 

satisfactory without 
improvement? 


Deficiencies 

warrant 

improvement 


Moderately objectionable 
deficiencies 


Adequate performance requires 
considerable pitot compensation 


Adequate performance requires extensive 
pitot compensation 


Very objectionable but 
tolerable deficiencies 


Is adequate \ 
performance attainable 
with a tolerable pitot 
K workload? 


Deficiencies 

require 

improvement 


Control wilt be lost during some portion of 
required operation 


Improvement 

mandatory 


it controllable? 


Deftnabon ol required ooerabdn «wotves desenabon or flight phase 
sncllcx subphasre; wifti nrxXTnpariying ccncfcbans 


Cooper-Harper Ref NASA TND &153 


Excellent 

Pilot compensation not a factor for 

Highly dcsireable 

desired performance 

Good 

Pilot compensation not a factor for 

Negligible deficiencies 

desired performance 

Fair-some mildly 

Minimal pilot compensation required for 

unpleasant deficiencies 

desired performance 


Major Deficiencies 

Adequate performance not attainable with 
maximum tolerable pilot compensation. 
Controllability not in question. 

Q 

Major Deficiencies 

Considerable pitot compensation is required 
for control 


Major Deficiencies 

Intense pilot compensation is required to 
retain control 

0 


Cooper/Harper HQR Rating Scale 


HQR 1. Excellent - Highly desirable. Pilot compensation not a factor for desired performance. 

HQR 2. Good - Negligible deficiencies. Pilot compensation not a factor for desired performance. 

HQR 3. Fair - Some mildly unpleasant deficiencies. Minimal pilot compensation required for 
desired performance. 

HQR 4. Minor deficiencies. Desired performance required considerable pilot compensation. 

HQR 5. Moderately objectionable deficiencies. Adequate performance requires considerable pilot 
compensation. 

HQR 6. Very objectionable but tolerable deficiencies. Adequate performance requires extensive 
pilot compensation. 

HQR 7. Major deficiencies. Adequate performance not obtainable with maximum pilot 
compensation. Controllability not in question. 
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HQR 8. Major deficiencies. Considerable pilot compensation is required for control. 

HQR 9. Major deficiencies. Intense pilot compensation is required to retain control. 

HQR 10. Major deficiencies. Control will be lost during some portion of required operation. 
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APPENDIX D. NASA-TLX RATING SHEET 


Subject Instructions: Weights 

Throughout this experiment the rating scales are used to assess your experiences in the different test 
conditions. Scales of this sort are extremely useful, but their utility suffers from the tendency people 
have to interpret them in individual ways. For example, some people feel that mental or temporal 
demands are the essential aspects of workload regardless of the effort they expended or the 
performance they achieved. Others feel that if they performed well the workload must have been 
low, and vice versa. Yet others feel that effort or feelings of frustration are the most important 
factors in workload, and so on. The results of previous studies have found every conceivable pattern 
of values. In addition, the factors that create levels of workload differ depending on the task. For 
example, some tasks might be difficult because they must be completed very quickly. Others may 
seem easy or hard because of the intensity of mental or physical effort required. Yet others feel 
difficult because they cannot be performed well, no matter how much effort is expended. 

The evaluation you are about to perform is a technique that has been developed by NASA to assess 
the relative importance of six factors in determining how much workload you experienced. The 
procedure is simple: you will be given a sheet of paper with all possible pairs (omitting duplications) 
of rating scale titles (e.g., Mental Demands vs. Effort), and asked to circle the item that was more 
important to your experience of workload in the tasks that you will be performing in the experiment. 
Circle only one of the two scale titles on each line — the scale title that represents the more important 
contributor to workload for the specific tasks you will be performing in this experiment. When you 
are done, one of the two words in each pair should be circled. 

After you have finished the entire sheet of pairs we will be able to use the pattern of your choices to 
create a weighted combination of the ratings from each test trial into a summary score for that trial. 
Please consider your choices carefully and make them consistent with how you will be using the 
rating scales during the particular task you were asked to evaluate. Don't think that there is any one 
correct pattern; we are only interested in your opinions. If you have any questions, please find the 
tester and ask them now. Thank you very much for your participation. 
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Name /ID: 

Maneuver completed: 
HQR 


WEIGHTING OF RATING SCALES 


Circle only one of the two scale titles on each line — the scale title that represents the more important 
contributor to workload for the tasks you will perform in the test. 


Performance 
Mental Demand 
Temporal Demand 
Physical Demand 
Mental Demand 
Performance 
Frustration 
Physical Demand 
Mental Demand 
Temporal Demand 
Mental Demand 
Temporal Demand 
Physical Demand 
Frustration 


Physical Demand 

Temporal Demand 

Effort 

Frustration 

Effort 

Effort 

Mental Demand 
Effort 

Performance 
Performance 
Physical Demand 
Frustration 
Temporal Demand 
Performance 


Effort 


Frustration 


63 



Subject ID: JoeSample Task ID: 1 

WORKLOAD and SITUATION AWARENESS RATING 
SHEET 


MENTAL DEMAND 1 1 1 1 1 1 1 1 1 1 1 1 1 |l 1 1 1 1 

J_U 

Very Low 

Very High 

PHYSICAL DEMAND 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

J_U 

Very Low 

Very High 

TEMPORAL DEMAND 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

1 i 1 

Very Low 

Very High 

PERFORMANCE 1 1 1 1 |l 1 1 1 1 1 1 1 1 1 1 1 1 1 

J_U 

Perfect 

Failur 

e 

EFFORT 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

LU 

Very Low 

Very High 

FRUSTRATION 1 1 1 1 1 | 1 1 1 1 1 1 1 1 1 1 1 1 

J_U 

Very Low 

Very High 

SITUATION AWARENESS: Knowledge of the interrelationships between 
others' [other aircraft or objects] and one's own position in time and 
in 3-dimensional space, such that one may operate in the best 
offensive and defensive manner possible. 


Rate your level of situation awareness in the mission you just flew. 

1 i 1 i 1 i 1 i 1 i i 1 i 1 i 1 i 

■LU 

Very Low 

®Very High 


Sample NASA-TLX Workload Rating Form 
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APPENDIX E. SUBJECTIVE QUESTIONNAIRE SCALES AND RESULTS 


The following items were rated utilizing the scale found in the adjacent column. Ratings were taken 
in an online format with mouse input on a slider scale with endpoints corresponding to those listed. 
Significant findings of variance as a result of symbology have been graphed. Selected ratings that 
showed a significant result were repeated in the main body of this report and analyzed to determine 
cause. 


1. Symbology clutter 

low-high 

2. Correct symbol positioning 

none-all 

3. Tendency to shift attention 

n eve r-al ways 

4. Ease of dividing/shifting attention 

easy-difficult 

5. Enhanced mission effectiveness 

enhanced-degradation 

6. Enhanced situation awareness 

enhanced-degradation 

11. Enhanced flight performance 

enhanced-degradation 

12. Enhanced mission performance 

enhanced-degradation 

13. Enhanced flight safety 

enhanced-degradation 

14. Aircraft roll control 

easy-difficult 

15. Aircraft yaw 

easy-difficult 

16. Aircraft pitch 

easy-difficult 

17. Airaft rate of climb/descent 

easy-difficult 

18. Achieve and maintain hover 

easy-difficult 

19. Aircraft attitude control 

easy-difficult 

20. Aircraft heading control 

easy-difficult 

21. Aircraft airspeed control 

easy-difficult 

22. Waypoint navigation performance 

easy-difficult 

24. Accurately reporting heading to 
target in tactical scenarios 

useless-useful 

25. Rapidly reporting heading to target 
in tactical scenarios 

useless-useful 

26. Spatial orientation in the world 

never-always 

27. Altitude above the terrain 

never-always 

28. Aircraft's indicated airspeed 

never-always 

30a. Readability-horizon reference line 

satisfactory-unsatisfactory 

30b. Readability-heading tape 

satisfactory-unsatisfactory 

30c. Readability-radar altitude 

satisfactory-unsatisfactory 

30d. Readability-airspeed 

satisfactory-unsatisfactory 

30e. Readability-barometric altitude 

satisfactory-unsatisfactory 

30f. Readability-vertical situation 
indicator 

satisfactory-unsatisfactory 

31a. Symbol Dynamics-horizon 
reference line 

satisfactory-unsatisfactory 

31b. Symbol Dynamics-heading tape 

satisfactory-unsatisfactory 

31c. Symbol Dynamics-radar altitude 

satisfactory-unsatisfactory 

31 d. Symbol Dynamics-airspeed 

satisfactory-unsatisfactory 

31 e. Symbol Dynamics-barometric 
altitude 

satisfactory-unsatisfactory 

31f. Symbol Dynamics-vertical 
situation indicator 

satisfactory-unsatisfactory 
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Legend for all graphs: 



0 20 40 60 80 100 


None < — > All 

Average rating of correct symbol position within a symbology set ( n = 6) 



0 20 40 60 80 100 


Unsatisfactory < — > Satisfactory 

Rating of horizon reference line readability for each symbology set (n = 6) 



0 20 40 60 80 100 


Difficult < — > Easy 

Rating of ease with which pilots could control aircraft pitch in = 6) 
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0 20 40 60 80 100 

Difficult < — > Easy 

Rating of ease with which pilots could control aircraft rate of climb and descent ( n = 6) 



0 20 40 60 80 100 


Unsatisfactory < — > Satisfactory 

Rating of satisfaction with the dynamics of the radar altimeter symbol ( n = 6) 



Rating of use of symbology for accurately and rapidly reporting heading to target information in the 
tactical scenarios in = 6) 



Rating of vertical situation indicator readability for each symbology set in = 6) 
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APPENDIX F. PILOT PERFORMANCE DATA 


This section contains all of the data collected during the experiment. Some data represent individual 
pilot performance and some charts represent averaged performance for all pilots. Data below that 
showed a significant result were repeated in the main body of the report and analyzed in detail. 


Comanche SIM II 
Test Results 
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PERFORMANCE ANALYSES 


Unusual Attitude Recovery (Color Code: Desired, Adequate, Not Adequate) 


Performance 

Parameter 

Compressed 

Contact 

Analog 


Significance 

Roll Angle 

2.60 deg 

2.94 deg 


no sig 

Pitch Angle 

3.14 deg 

2.90 deg 


no sig 

Vertical Rate 

160.64 fpm 

116.90 fpm 


no sig 

Recovery Time 

20.75 sec 

19.47 sec 


sig Trial 
effect p < .05 


Hover Turn Bob-up (Position Hold) 


Performance 

Parameter 

Compressed 

Contact 

Analog 


Significance 

Longitudinal 

Error 

5.47 ft. 

4.99 ft. 


no sig 

Lateral Error 

4.89 ft. 

6.29 ft. 


sig Trial 
effect p < .05 

Final Rad Alt 
Error 

1 .77 ft. 

2.42 ft. 


no sig 

Heading Error 

1 .45 deg 

1 .66 deg 


no sig 

Time to 
Complete 

13.35 sec 

15.37 sec. 


sig Symbol 
effect p < .01 


Hover Turn Bob-up (AFCS) 


Performance 

Parameter 

Compressed 

Contact 

Analog 


Significance 

Longitudinal 

Error 

28.27 ft. 

37.36 ft. 


no sig 

Lateral Error 

22.29 ft. 

26.29 ft. 


no sig 

Final Rad Alt 
Error 

2.80 ft. 

3.91 ft. 


no sig 

Heading Error 

2.09 deg 

1 .59 deg 


sig Trial 
effect p < .05 

Time to 
Complete 

16.98 sec. 

18.37 sec. 


no sig 
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Mean Recovery Time 


UNUSUAL ATTITUDE PERFORMANCE DATA 


Time to Recover 



2.00 

0.00 


Compressed 1 Contact Analog 1 

Compressed 2 Contact Analog 2 


Symbology/Block 



Trial Number 
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VSI Rate at Recovery 


200.00 



Compressed Contact Analog 

Symbology 


Final Pitch Angle 


4.00 



Compressed Contact Analog 

Symbology 
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Final Roll Angle 


a:i 

L 

~o 

£= 

<D 

a:i 

£= 

< 

"o 

LtL 

£= 

ro 

(D 



Compressed 1 Contact Analog 1 

Compressed 2 Contact Analog 2 


Symbology/Block 
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Mean Time to Complete (sec) 


HOVER TURN BOB-UP (POSITION HOLD) PERFORMANCE DATA 

Time to Complete x Pilot ID 


20.00 



O 

0 

CD 

+-• 

0 

Q_ 

E 

o 

O 

o 

-i— ' 

0 

E 


c 



Pilot ID 

Time to Complete 



Compressed Contact Analog 


Symbology 
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Mean Lateral Excursion (ft) 


Heading Error 


Qi 

g 

lT3 

<D 


lT3 

0 > 



Compressed Contact Analog 


Symbology 


Lateral Error 



Compressed Contact Analog 


Symbology 


Longitudinal Error 



Symbology 


Final RADALT Error 

4.00 

3.50 1 



Compressed Contact Analog 


Symbology 
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HOVER TURN BOB-UP (AFCS) PERFORMANCE DATA 


Time to Complete 



Symbology 


3.00 


Heading Error 



Compressed 


Contact Analog 


Symbology 
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30.00 


Lateral Error 


25.00 


c 20.00 

o 

'cn 

to 

o 

UJ 15.00 

TO 

0 

TO 


c 

TO 

0 


10.00 


5.00 


0.00 1 



Compressed Contact Analog 

Symbology 


to 

o 

C/) 

i_ 

TO 

o 

X 

HI 

"to 

TO 


O) 

to 

o 

_l 

TO 

TO 

0 


Longitudinal Error 



Compressed Contact Analog 


Symbology 


78 


Mean RadAlt Error (ft) 


RADALT Error 



Compressed Contact Analog 


Symbology 


NOE MISSION PERFORMANCE DATA 


Azimuth to Target Estimation Error 


O) 

0 

T3 


2 

S - 

LD 


0 

E 

to 

LU 

U) 


"O 

0 


18.00 

16.00 

14.00 

12.00 
10.00 

8.00 

6.00 

4.00 

2.00 
0.00 



Compressed Contact Analog 


Symbology 


Time to Call Azimuth to Target (speed) 



Compressed 


Contact Analog 


Symbology 
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HOVER TURN BOB-UP (POSITION HOLD/AFCS) HQRS 



Compressed AFCS Contact Analog AFCS 

Symbology/Flight Mode 


WORKLOAD AND SITUATIONAL AWARENESS RATINGS 

Overall Workload 


Overall Workload across Symbology Sets 



□ Overall Workload 


Symbology Sets 
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NASA-TLX Scores for All Maneuvers Completed 


TLX Scores for Symbology Sets across Task Types 



□ Compressed 
■ Contact Analog 


Task Types/Maneuvers 


Situational Awareness Ratings 


Overall Situation Awareness across Symbology Sets 



□ Situation Awareness 


Symbology Sets 


Situation Awareness for Symbology Sets across Task Types 


100.00 

90.00 

80.00 

70.00 

60.00 
SS 50.00 

40.00 

30.00 

20.00 

10.00 

0.00 

BU UA CON NOE 



□ Compressed 
■ Contact Analog 


Tasks/Maneuvers 
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APPENDIX G. PILOT COMMENTS 


Each pilot was ask to provide an end of simulation top-level summary of their overall opinion 
concerning the effectiveness and usability of the two different symbology sets. Their comments are 
provided below. The summary comments are followed by the detailed comments that explained the 
HQR ratings for the hover bob-up maneuver. 

Pilot 1 

1. Consider having the pilot look off axis during the Unusual Attitude Recovery. 

2. Fog was an excellent and realistic addition to the UA. 

3. Favored the compressed heading tape for 2 reasons. First the tape was more readable if a rapid 
turn. Second, you could see the target heading during the entire maneuver, which reduced cross- 
check time and workload. 

4. Contact Analog symbology is not readable during rapid turn or rapid head movements. 

5. Favored the VSI tape in 1295 symbology. It seemed easier to use with the pointer against the 
scale. You could set the pointer on the index and use appropriate control input to keep it where 
you put it. 

6. Liked the hybrid #2 symbol set. Would like to add the 1295 VSI tape to it. 

7. If I had to choose between CA and 1295 I would choose CA. 

8. The earth-referenced horizon line disappears during certain nose up or head up situations. Not 
having a visible horizon could present an unsafe situation. 

9. The symbology seems too spread out. This increases mental workload in the cross check. 
Moving the symbology closer to the center of the display would reduce workload. 

10. The null point on the 1295 symbology VSI is not predominant enough causing you to have to 
search for it. 

Pilots 2 & 5 

1. If you had to choose one symbology set in its entirety which did you prefer? 

Pilot 5 - Hybrid #2 - Contact Analog with compressed heading tape. 

Pilot 2 - 1295. 

2. Did you see any safety issues with any of the symbology sets? 

- Barometric altimeter was too far to the left of the central field of view to be usable. 

- Earth referenced horizon line could not be seen during nose up or nose down pitch attitudes 
such as those experienced during the unusual-attitude task. 
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3. Were there any improvements needed to make Contact Analog symbology more usable or 
mission effective? 

- Move the torque symbol to the left of the radar altimeter. 

- The first 100 ft of the radar altimeter should have tic marks at 10-ft increments. 

- Moving the altitude information to the left and airspeed to the right was not a problem. 
Adaptation was easy. 

- Replace the Contact Analog VSI with a pointer against a scale like the 1295 presentation. 
Move the entire 1295 altitude group in to replace the Contact Analog. 

- Replace the torque thermometer symbol with a digital readout that is coded to show OGE 
hover torque as well as max continuous and max torque available. 

- The head-referenced heading tape has no known tactical value. Bearing to target information 
can be gotten through other means other than pointing the head and reading the scale. 
Recommend the compressed heading tape with a digital readout beneath the scale. 

- Would like to be able to adjust the intensity of the symbology to compensate for the variable 
brightness of the outside scene. 

Pilots 3 & 4 

Optimum Display Preferences and Justification following Comanche Heading Tape vs. 1295 
Symbology Simulation 

General Comments 

1. I saw little need for the symbology to be spread to the edges of the display. I had difficulty 
focusing on images on the edges of the display in the contact analog format requiring to scan 
away from the central field of view especially as the task requirements were increased. 

2. If symbology is moved to the edges of the display, it must be bright enough to be interpreted by 
the pilot without requiring scan and focus during demanding multi-axis tasks. 

3. Airspeed and altitude in the form of a standard T or alternate had little impact on the workload or 
task performance during the simulation. 

4. Unusual attitude recovery tasks were independent of symbology tasks once task performance 
was achieved. Desired performance was achieved using basic attitude and pitch corrections only 
until the acceleration ball could be driven to the center of the 5 -knot hover cueing dots. IV VI 
indications were also very important during the tasks to control climb and descent. 

5. Little difference in task performance was observed with any of the hybrid symbology sets. 

6. Skid/slip ball should not appear in any symbology set unless turn coordination logic is ON. 

7. A compromised set of symbology from each symbology set can mitigate safety issues and 
substantially improve the performance of both display sets for the maneuvers evaluated during 
the simulation. 
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8. Preferred Symbology Set - 1295 

Justification: Nearly all of the symbology was focused on primary field of view, was easy to read 
and interpret, and symbology dynamics were predictable. 

Recommended Changes 

a. Add digits at the center of the heading tape to provide the pilot with the direction he is looking. 
The 180-deg field of view on the compressed heading tape provides a very predictable indication 
of the aircraft heading in a screen-fixed method. This digital indication will help the pilot with 
target identification in azimuth without cluttering the center of the pilot FOV. 

b. Remove the LOS indication in preference to an aircraft symbol that is implemented in the same 
manner as the Contact Analog Comanche symbology. The LOS indicator only provides pitch 
cueing while the aircraft symbol used in the Contact Analog symbology provides pitch cueing as 
well as a reference to the pilot as to the direction the aircraft nose is pointing. The location of the 
nose helps with sideslip control at slow speeds to help keep the tail behind the aircraft. The 
aircraft symbol also provides a quick reference to the pilot when he is looking over the nose. 

This indication is currently missing in the Apache 1295 symbology. 

c. Add the Contact Analog current waypoint and next waypoint indications to the 1295 Apache 
symbology set, but reduce their size. This symbology is very useful and easy to interpret over the 
digital information provided in the Apache 1295 symbology set. Retain the command heading 
indicator on the heading tape for cross reference. 

Safety Issues 

a. Pilot is not always aware of the direction he is looking, which can result in spatial disorientation 
especially during more aggressive maneuvering. This is especially true during unusual attitudes 
when inner ear disorientation can result due to the head being in a different axis than the visual 
indications interpreted by the pilot. 

b. Pilot is not always aware of the location of the nose of the aircraft relative to the flightpath 
vector, which can result in poor turn coordination at low airspeed during NOE flight. This may 
result in tree contact or potential rollover if the pilot thinks he is landing straight ahead and the 
nose is not in proper alignment. 

c. Navigation indications were difficult to follow and provided poor cueing to the pilot on the 
visual display. The use of a moving map or hand-held map is required to provide actual 
identification of the waypoint location resulting in additional workload. 

Contact Analog Modifications 

Recommended Changes 

d. Adopt the 1295 heading tape with digital heading indicating the direction the pilot is looking. 

The Contact Analog heading tape is difficult to use when precision and aggressive maneuvering 
is required. The pilot has reduced situational awareness at high yaw rates. 

e. Remove the IVVI, barometric altitude, and radar altitude indications including the 6-sec 
predictor in favor of the Apache 1295 barometric altitude, radar altitude, and IYYI indications. 


85 



The 1295 indications could be made brighter and still moved to the outer edges of the display 
whether they are on the right or left side of the display. 

f. Remove the torque indication in favor of digital torque indication with various shadings or 
colorings to indicate torque limits. This indication was of limited utility and can be implemented 
in a smaller amount of space while providing similar clarity to the pilot. The current symbology 
uses critical real estate that is better used for barometric and radar altitude information rather 
than the very bright torque indication. 

g. Reduce the size of the waypoint markers. These were very effective for navigation and 
identifying the waypoint location in the scene. 

h. Adopt the screen-fixed artificial horizon line used in the 1295 symbology set. Easier to use when 
it remains in the primary field of view and prevents clutter at the top and bottom of the display. 

i. Vertical axis control was difficult due to the poor predictability of the 6-sec predictor and the 
poor dynamics positioning of the IVVI. The IVVI was too dim and too far outside the primary 
field of view for the pilot to use effectively at low airspeed. These issues resulted in more ground 
contacts with Contact Analog than with any other display method. 

j. Earth-referenced symbology can cause significant screen clutter depending on where the pilot is 
looking. This can result in display confusion and potential disorientation. 

k. The Earth-referenced horizon line was difficult to use with respect to unusual attitudes and could 
not be present in the field of view depending on where the pilot was looking. This can result in 
significant disorientation especially during the initial phases of an unusual attitude recovery. 

l. The significant gaps in the heading tape and the Earth-fixed implementation of the heading tape 
made it difficult to interpret especially at high yaw rates or while trying to manage a multi-axis 
task. This increase in workload in conjunction with poor vertical cueing can significantly impact 
pilot performance during low, moderate, and especially high-gain multi-axis tasks. 

Pilot 6 

1. In order to evaluate the hybrids more fully you need to incorporate NOE and contour maneuvers 
into the evaluation. 

2. Training locally was adequate when combined with the 7 hrs of CPC time prior to coming to 
Ames. Without the CPC time there would need to be more simulation hours allotted for training. 

3. Which symbology set would you choose if you had to choose one without modifying it? Answer: 
Would choose 1295 over Contact Analog. 

4. Were there any safety issues noted with any of the symbology sets? 

- The variable radar altimeter scaling may cause inadvertent ground impact due to vertical rate 
buildup. 

5. Were there any design changes that you could recommend for Contact Analog symbology? 

- Recommend the radar altimeter have 10-ft tic mark scaling up to 100-ft AGL. 

- Recommend the symbology be moved in toward the central FOV. The current design is too 
spread out, which increases workload and slows the cross check. 


86 



- Switching altitude information to the right side and moving airspeed to the left was not an 
issue and seemed to make sense with respect to which hand controls what axis. 

- During the unusual attitude, when the airspeed increased to > 20 kts, the trim ball appeared 
and it was mistaken for the acceleration cue. 

6. Wrist coupling was unavoidable while trying to maintain hover position with the SAC. 

If you could design a symbology set for the maneuvers in this test what would it look like? 

- Compressed heading tape 

- Indexed VS I with 1295 style pointer against a fixed scale 

- Linear radar altitude scale with multiple regions more like the 1295 scale 

- Contact Analog torque scale as designed (digital readout with ring and index marks) 

- Contact Analog hover cue symbology 

- Move the symbology in closer to the central FOV 

- Delete the 18° gap in the horizon line. Make the gap fit the iron wings 

- Need a selectable pitch ladder for up-and-away flight 

- Display a hover position reference 

- Display an analog airspeed indicator 

- Put a filter on the radar altimeter 6-sec predictor to make it more usable. The filter should act 
much like the Hybrid 3 predictor. 
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